Oct 29 21:39:20.755979 ip-10-0-128-100 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 21:39:20.755989 ip-10-0-128-100 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 21:39:20.755997 ip-10-0-128-100 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 21:39:20.756295 ip-10-0-128-100 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 21:39:31.005312 ip-10-0-128-100 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 29 21:39:31.005334 ip-10-0-128-100 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ca15e9674c3e4088af4a29adfeb85284 -- Oct 29 21:41:34.982079 ip-10-0-128-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:41:35.397624 ip-10-0-128-100 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:35.397624 ip-10-0-128-100 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:41:35.397624 ip-10-0-128-100 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:35.397624 ip-10-0-128-100 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:35.400625 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400562 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.400625 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400625 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.400625 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400629 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400632 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400635 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400638 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400640 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400643 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400646 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400648 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400652 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400654 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400657 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400660 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400662 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400665 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400667 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400670 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400672 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400675 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400678 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400680 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.400712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400688 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400691 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400695 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400699 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400702 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400704 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400707 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400709 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400712 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400715 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400717 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400720 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400723 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400726 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400728 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400730 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400733 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400736 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400738 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.401191 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400740 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400743 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400745 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400749 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400753 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400756 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400759 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400762 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400765 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400767 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400770 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400773 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400777 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400779 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400786 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400790 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400793 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400795 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400798 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.401637 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400800 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400803 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400805 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400808 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400824 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400827 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400829 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400832 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400834 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400837 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400840 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400842 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400844 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400847 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400849 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400852 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400854 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400857 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400859 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400862 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.402150 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400864 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400867 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400869 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400872 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400874 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400877 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400879 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400882 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400890 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400893 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400896 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400898 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400901 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400903 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400906 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400908 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400911 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400913 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400916 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400919 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.402621 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400921 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400923 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400926 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400928 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400931 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.400935 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401687 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401693 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401696 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401699 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401702 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401705 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401707 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401710 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401712 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401715 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401718 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401721 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401724 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.403137 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401726 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401729 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401732 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401735 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401738 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401740 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401743 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401745 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401748 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401750 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401753 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401755 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401759 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401763 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401767 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401770 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401776 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401779 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401782 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.403598 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401784 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401787 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401789 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401792 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401794 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401797 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401800 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401802 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401805 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401807 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401810 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401828 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401831 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401833 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401836 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401839 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401842 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401844 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401847 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401849 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.404095 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401852 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401854 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401857 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401859 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401862 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401864 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401867 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401869 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401872 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401875 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401878 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401880 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401883 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401885 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401888 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401890 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401893 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401895 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401898 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401901 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.404561 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401903 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401906 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401909 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401911 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401914 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401917 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401919 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401922 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401925 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401928 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401930 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401933 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401935 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401937 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401940 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401942 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401945 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401947 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401950 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401952 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.405112 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401955 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401957 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401960 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401962 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401965 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401967 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401970 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401972 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401975 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401977 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401981 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401986 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.401989 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402450 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402455 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402458 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402460 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402463 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402466 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.405591 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402468 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402471 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402474 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402477 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402479 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402482 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402484 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402487 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402489 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402492 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402494 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402497 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402500 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402503 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402507 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402510 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402512 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402515 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402517 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402520 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.406334 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402523 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402526 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402528 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402531 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402534 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402536 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402539 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402541 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402544 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402546 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402549 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402551 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402553 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402557 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402563 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402567 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402570 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402572 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402575 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.407193 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402577 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402580 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402582 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402585 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402588 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402590 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402593 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402596 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402599 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402601 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402605 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402608 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402611 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402614 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402616 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402619 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402621 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402624 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402626 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402629 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.407712 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402631 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402634 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402637 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402639 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402642 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402644 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402647 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402651 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402654 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402657 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402659 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402662 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402664 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402667 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402669 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402672 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402674 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402676 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402679 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.408369 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402682 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402685 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402687 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402690 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402692 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402694 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402697 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402700 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402703 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402705 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402708 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402710 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402712 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402715 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402718 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402720 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402723 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402725 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402727 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402730 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.408831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402733 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.402736 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404115 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404125 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404132 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404136 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404140 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404143 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404150 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404153 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404156 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404159 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404163 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404166 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404169 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404172 2574 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404175 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404178 2574 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404181 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404184 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404188 2574 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404191 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404194 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404197 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:41:35.409391 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404200 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404204 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404207 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404210 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404213 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404216 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404219 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404222 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404226 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404229 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404233 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404237 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404240 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404243 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404246 2574 flags.go:64] FLAG: --enable-server="true" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404248 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404253 2574 flags.go:64] FLAG: --event-burst="100" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404256 2574 flags.go:64] FLAG: --event-qps="50" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404259 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404262 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404265 2574 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404269 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404272 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404275 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404277 2574 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:41:35.409984 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404280 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404283 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404286 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404289 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404292 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404295 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404298 2574 flags.go:64] FLAG: --feature-gates="" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404302 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404305 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404308 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404311 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404313 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404316 2574 flags.go:64] FLAG: --help="false" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404319 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404322 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404327 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404330 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404333 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404336 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404340 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404342 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404345 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404348 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:41:35.410626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404351 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404354 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404356 2574 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404359 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404362 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404365 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404368 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404370 2574 flags.go:64] FLAG: --lock-file="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404373 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404376 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404379 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404384 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404386 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404389 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404392 2574 flags.go:64] FLAG: --logging-format="text" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404395 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404399 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404402 2574 flags.go:64] FLAG: --manifest-url="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404404 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404408 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404411 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404415 2574 flags.go:64] FLAG: --max-pods="110" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404418 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404421 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404425 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:41:35.411197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404427 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404430 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404433 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404436 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404443 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404446 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404450 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404452 2574 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404455 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404458 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404461 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404465 2574 flags.go:64] FLAG: --port="10250" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404468 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404471 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0728368544a028c6d" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404474 2574 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404477 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404480 2574 flags.go:64] FLAG: --register-node="true" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404482 2574 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404486 2574 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404489 2574 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404491 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404494 2574 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404498 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404501 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404504 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404506 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:41:35.411971 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404509 2574 flags.go:64] FLAG: --runonce="false" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404512 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404515 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404518 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404521 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404525 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404528 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404531 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404534 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404536 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404539 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404542 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404545 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404548 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404551 2574 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404553 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404558 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404563 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404565 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404569 2574 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404572 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404575 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404578 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404580 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404583 2574 flags.go:64] FLAG: --v="2" Oct 29 21:41:35.412580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404587 2574 flags.go:64] FLAG: --version="false" Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404591 2574 flags.go:64] FLAG: --vmodule="" Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404595 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.404598 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405720 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405730 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405734 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405741 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405745 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405748 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405751 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405754 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405757 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.405843 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406002 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406008 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406014 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406019 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406024 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406028 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.413198 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406036 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406046 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406051 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406056 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406060 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406065 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406069 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406073 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406077 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406082 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406086 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406091 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406095 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406100 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406104 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406108 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406113 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406122 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406126 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406131 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.413661 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406136 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406140 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406144 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406148 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406152 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406157 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406164 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406168 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406173 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406178 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406182 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406187 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406191 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406196 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406200 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406204 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406208 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406213 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406217 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406222 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.414161 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406227 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406231 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406235 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406240 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406244 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406248 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406252 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406257 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406262 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406266 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406270 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406275 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406279 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406283 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406287 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406292 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406296 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406301 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406307 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.414630 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406312 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406316 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406321 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406325 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406329 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406333 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406337 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406342 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406347 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406353 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406359 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406363 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406368 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406373 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406378 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406383 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406387 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406392 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406396 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.415135 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406400 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406404 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406409 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406413 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406417 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406422 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406426 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406430 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406434 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406439 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406443 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406447 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.406497 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406559 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406568 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406574 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.415568 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406579 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406583 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406587 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406592 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406596 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406600 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406606 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406611 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406616 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406620 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406624 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406628 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406632 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406636 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406640 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406645 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406649 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406653 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406658 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.415973 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406662 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406667 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406671 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406675 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406679 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406684 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406688 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406692 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406698 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406704 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406711 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406715 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406720 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406724 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406729 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406733 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406737 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406742 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406746 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.416421 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406750 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406756 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406761 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406765 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406770 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406774 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406778 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406783 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406787 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406793 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406797 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406801 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406806 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406810 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406834 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406838 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406842 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406847 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406851 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406855 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.417506 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406859 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406863 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406868 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406874 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406877 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406882 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406886 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406890 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406897 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406903 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406909 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406914 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406918 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406923 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406928 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406932 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406936 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406941 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406945 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406949 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.417991 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406953 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406957 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406961 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406966 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406970 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406974 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406978 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406983 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406987 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406991 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406995 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.406999 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407003 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407007 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407011 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407022 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407027 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407033 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.418470 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407037 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407041 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407045 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407049 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407053 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.407058 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.413248 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.413261 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.413276 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.413281 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413332 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413336 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413339 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413342 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413346 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413349 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413352 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413354 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413357 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413359 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.418933 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413362 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413364 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413367 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413370 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413372 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413375 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413378 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413380 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413383 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413385 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413388 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413391 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413395 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413398 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413401 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413404 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413407 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413409 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413412 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.419400 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413415 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413418 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413421 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413424 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413426 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413429 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413431 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413435 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413439 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413442 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413446 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413449 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413451 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413454 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413457 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413459 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413462 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413464 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413467 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.419884 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413469 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413471 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413474 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413476 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413479 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413481 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413484 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413487 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413489 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413492 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413494 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413497 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413499 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413502 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413505 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413508 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413510 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413513 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413515 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413518 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.420343 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413520 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413523 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413525 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413528 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413530 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413533 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413535 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413538 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413540 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413543 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413545 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413547 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413550 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413552 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413555 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413557 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413559 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413563 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413565 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.420864 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413567 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413570 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413572 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413575 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413577 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413580 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413582 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413585 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413588 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413590 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413593 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413596 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413598 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413601 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413603 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413606 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413609 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413611 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.421298 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413614 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.413641 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413779 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413784 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413787 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413790 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413792 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413795 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413797 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413800 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413802 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413805 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413808 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413810 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413832 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413834 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.421713 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413837 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413839 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413842 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413844 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413847 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413850 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413852 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413855 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413858 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413860 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413863 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413866 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413868 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413871 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413873 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413876 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413878 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413881 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413883 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.422221 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413886 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413888 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413890 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413893 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413896 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413901 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413904 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413907 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413909 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413912 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413914 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413917 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413920 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413922 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413925 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413927 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413930 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413932 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413935 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413937 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.422683 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413940 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413942 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413945 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413947 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413950 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413952 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413954 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413957 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413959 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413961 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413964 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413966 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413968 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413971 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413973 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413975 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413978 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413980 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413982 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413985 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.423225 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413987 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413990 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413992 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413994 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413997 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.413999 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414001 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414004 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414006 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414008 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414011 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414013 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414016 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414018 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414021 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414024 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414026 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414029 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414032 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.423708 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414035 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414038 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414041 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414043 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414046 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414048 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414050 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414053 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414055 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414058 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414061 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414063 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414066 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414068 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.414091 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414121 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:35.424192 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414125 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414128 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414131 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414133 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414136 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414138 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414141 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414143 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414145 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414148 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414150 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414152 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414155 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414158 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414161 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414163 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414166 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414169 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414172 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:35.424613 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414174 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414177 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414179 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414182 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414184 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414186 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414189 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414191 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414194 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414196 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414198 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414204 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414208 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414212 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414215 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414218 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414221 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414223 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414226 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414228 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:35.425139 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414231 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414233 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414236 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414239 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414241 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414244 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414247 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414251 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414253 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414256 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414259 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414261 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414264 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414266 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414269 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414271 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414273 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414276 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414278 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414281 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:35.425687 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414283 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414286 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414288 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414293 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414295 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414298 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414300 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414302 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414305 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414307 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414310 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414312 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414315 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414317 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414320 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414323 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414325 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414327 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414330 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414334 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:35.426216 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414337 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414339 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414342 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414345 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414347 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414351 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414354 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414357 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414360 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414362 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414365 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414367 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414370 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414372 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414374 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414379 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414381 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414385 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414387 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:35.426723 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414390 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414392 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414395 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414397 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414399 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414402 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:35.414404 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.414454 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.416328 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.417134 2574 server.go:1014] "Starting client certificate rotation" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.417229 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.418027 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.419312 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:35.427246 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.420329 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:35.432357 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.432338 2574 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:41:35.435589 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.435573 2574 log.go:25] "Validated CRI v1 image API" Oct 29 21:41:35.436788 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.436757 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:41:35.439240 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.439222 2574 fs.go:135] Filesystem UUIDs: map[220c75d5-1746-4dec-8987-ed758625d6a6:/dev/nvme0n1p4 5ed825ee-1838-494a-837b-a5195f63849e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 29 21:41:35.439302 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.439239 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 21:41:35.444105 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.443953 2574 manager.go:217] Machine: {Timestamp:2026-10-29 21:41:35.442599614 +0000 UTC m=+0.342087068 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101102 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a2da4e5ab56d77ec34e42d6cc122 SystemUUID:ec26a2da-4e5a-b56d-77ec-34e42d6cc122 BootID:ca15e967-4c3e-4088-af4a-29adfeb85284 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:b3:6b:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:b3:6b:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:86:b0:04:59:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:41:35.444975 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.444964 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:41:35.445053 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.445036 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:41:35.448615 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448593 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:41:35.448735 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448617 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:41:35.448782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448738 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:41:35.448782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448746 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:41:35.448854 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448789 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:35.448854 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.448802 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:35.449613 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.449603 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:41:35.454154 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454142 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:41:35.454276 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454267 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:41:35.454968 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454957 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:41:35.454997 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454972 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:41:35.454997 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454982 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:41:35.454997 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454991 2574 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:41:35.455079 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.454999 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:41:35.455304 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.455291 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:35.457583 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.457567 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:41:35.462723 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.462708 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:35.464095 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464080 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:41:35.464140 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464130 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:41:35.464326 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464315 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:41:35.464372 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464333 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:41:35.464372 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464343 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:41:35.464372 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464353 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:41:35.464372 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464362 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:41:35.464372 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.464372 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:41:35.465297 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465286 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:41:35.465334 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465300 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:41:35.465334 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465308 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:41:35.465334 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465321 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:41:35.465334 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465334 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:41:35.465500 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.465344 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:41:35.466729 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.466719 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:41:35.466761 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.466733 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:41:35.470349 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.470336 2574 server.go:1263] "Started kubelet" Oct 29 21:41:35.470438 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.470409 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:41:35.470580 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.470520 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:41:35.470661 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.470595 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:41:35.471598 ip-10-0-128-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:41:35.474851 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.474827 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:41:35.475744 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.475722 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:41:35.476465 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.476447 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 21:41:35.479506 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.479489 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:35.479987 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.479968 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:41:35.480118 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.480097 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.480523 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480503 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:41:35.480523 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480504 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:41:35.480648 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480537 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:41:35.480648 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480616 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:41:35.480648 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480636 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:41:35.480931 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480915 2574 factory.go:55] Registering systemd factory Oct 29 21:41:35.480931 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.480932 2574 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:41:35.481128 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481113 2574 factory.go:153] Registering CRI-O factory Oct 29 21:41:35.481128 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481129 2574 factory.go:223] Registration of the crio container factory successfully Oct 29 21:41:35.481216 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481165 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:41:35.481216 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481185 2574 factory.go:103] Registering Raw factory Oct 29 21:41:35.481216 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481197 2574 manager.go:1199] Started watching for new ooms in manager Oct 29 21:41:35.481974 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.481960 2574 manager.go:319] Starting recovery of all containers Oct 29 21:41:35.489535 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.489493 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:41:35.491087 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.491071 2574 manager.go:324] Recovery completed Oct 29 21:41:35.495517 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.495505 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.497046 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.497031 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.497096 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.497060 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.497096 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.497075 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.498469 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.498451 2574 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:41:35.498469 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.498467 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:41:35.498595 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.498488 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:41:35.498809 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.498770 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.499796 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.499777 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.499892 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.499832 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.499892 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.499849 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.500688 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.500676 2574 policy_none.go:50] "Start" Oct 29 21:41:35.500744 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.500699 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:41:35.500744 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.500710 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:41:35.502715 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.502701 2574 policy_none.go:44] "Start" Oct 29 21:41:35.505479 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.505456 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 21:41:35.505479 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.505473 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 21:41:35.506917 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.505658 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-100.ec2.internal.18e31e07cc9f370e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:35.47018011 +0000 UTC m=+0.369667562,LastTimestamp:2026-10-29 21:41:35.47018011 +0000 UTC m=+0.369667562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-100.ec2.internal,}" Oct 29 21:41:35.521966 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.521944 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:35.522877 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.522856 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 21:41:35.522977 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.522807 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-100.ec2.internal.18e31e07ce3927e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:35.497045985 +0000 UTC m=+0.396533441,LastTimestamp:2026-10-29 21:41:35.497045985 +0000 UTC m=+0.396533441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-100.ec2.internal,}" Oct 29 21:41:35.522977 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.522943 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 21:41:35.523092 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.523065 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 21:41:35.542381 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.542247 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:41:35.548488 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.548472 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lhj8x" Oct 29 21:41:35.555788 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.555706 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-100.ec2.internal.18e31e07ce397af9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-100.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:35.497067257 +0000 UTC m=+0.396554711,LastTimestamp:2026-10-29 21:41:35.497067257 +0000 UTC m=+0.396554711,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-100.ec2.internal,}" Oct 29 21:41:35.556994 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.556979 2574 manager.go:341] "Starting Device Plugin manager" Oct 29 21:41:35.557065 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.557009 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:41:35.557065 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557021 2574 server.go:87] "Starting device plugin registration server" Oct 29 21:41:35.557240 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557225 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:41:35.557304 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557239 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:41:35.557359 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557347 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:41:35.557447 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557426 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:41:35.557447 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.557441 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:41:35.557930 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.557907 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:41:35.558025 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.557961 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.565058 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.565039 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lhj8x" Oct 29 21:41:35.572981 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.572872 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-100.ec2.internal.18e31e07ce39aa95 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-100.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-100.ec2.internal,},FirstTimestamp:2026-10-29 21:41:35.497079445 +0000 UTC m=+0.396566901,LastTimestamp:2026-10-29 21:41:35.497079445 +0000 UTC m=+0.396566901,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-100.ec2.internal,}" Oct 29 21:41:35.624086 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.624065 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:41:35.624086 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.624088 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:41:35.625925 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.624107 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:41:35.625925 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.624166 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 21:41:35.640869 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.640850 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:35.657678 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.657639 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.658332 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.658308 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.658332 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.658334 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.658454 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.658343 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.658454 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.658362 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.678693 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.678675 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.678741 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.678694 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-100.ec2.internal\": node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.720768 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.720745 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.724475 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.724459 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal"] Oct 29 21:41:35.724532 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.724506 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.725239 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.725224 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.725324 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.725250 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.725324 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.725264 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.727479 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.727463 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.727644 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.727621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.727692 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.727658 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.728203 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728188 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.728278 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728212 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.728278 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728224 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.728278 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728250 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.728278 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728266 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.728278 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.728279 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.730261 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.730247 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.730338 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.730270 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:35.730939 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.730925 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:35.731037 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.730953 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:35.731037 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.730971 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:35.762370 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.762353 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-100.ec2.internal\" not found" node="ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.765253 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.765236 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-100.ec2.internal\" not found" node="ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.820866 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.820846 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.882034 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.882010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.882034 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.882035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.882175 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.882050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.921684 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:35.921616 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:35.983030 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.983030 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.983198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.983198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983105 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.983198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:35.983198 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:35.983138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:36.022531 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:36.022507 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:36.066061 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.066039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:36.068209 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.068188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:36.123265 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:36.123243 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:36.223790 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:36.223735 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:36.324208 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:36.324182 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:36.417430 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.417409 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 21:41:36.417887 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.417587 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 21:41:36.424295 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:36.424274 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:41:36.483641 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.483577 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:36.483641 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.483600 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:36.569048 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:36.569021 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2787fae5ca7f0b39f010401de9cba34b.slice/crio-eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959 WatchSource:0}: Error finding container eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959: Status 404 returned error can't find the container with id eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959 Oct 29 21:41:36.573423 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.573397 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:37 +0000 UTC" deadline="2028-06-10 03:37:46.808568846 +0000 UTC" Oct 29 21:41:36.573423 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.573420 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14141h56m10.235151115s" Oct 29 21:41:36.575333 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.575319 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:41:36.580323 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.580304 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:41:36.603517 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.603497 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:36.603985 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:36.603961 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186cd4ee51f579a3597049aa602c4c28.slice/crio-a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c WatchSource:0}: Error finding container a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c: Status 404 returned error can't find the container with id a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c Oct 29 21:41:36.604708 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.604692 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:41:36.625183 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.625163 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:36.626910 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.626877 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerStarted","Data":"a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c"} Oct 29 21:41:36.627901 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:36.627872 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" event={"ID":"2787fae5ca7f0b39f010401de9cba34b","Type":"ContainerStarted","Data":"eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959"} Oct 29 21:41:37.433153 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.433110 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:37.456487 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.456259 2574 apiserver.go:52] "Watching apiserver" Oct 29 21:41:37.479955 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.479922 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:37.503271 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.503248 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:41:37.503653 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.503621 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95","openshift-dns/node-resolver-2k6br","openshift-image-registry/node-ca-wkxjk","openshift-multus/multus-additional-cni-plugins-htxp7","openshift-multus/network-metrics-daemon-jvnws","openshift-network-diagnostics/network-check-target-2thvl","kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal","openshift-multus/multus-wpkwp","openshift-ovn-kubernetes/ovnkube-node-jj7md","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal","openshift-network-operator/iptables-alerter-8j8nv","kube-system/global-pull-secret-syncer-2tdpb","kube-system/konnectivity-agent-znvhf","openshift-cluster-node-tuning-operator/tuned-58pn4"] Oct 29 21:41:37.506167 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.506144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:37.508454 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.508436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:37.510454 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.510428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:37.512612 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.512589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:37.514788 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.514768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:37.515262 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:37.514878 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:37.517873 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.517056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:37.517873 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:37.517127 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:37.519239 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.519220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:37.520577 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.520555 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:41:37.520915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.520899 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:41:37.521008 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.520950 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:41:37.521701 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.521445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpkwp" Oct 29 21:41:37.521924 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.521903 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:41:37.522018 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.522004 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:41:37.522189 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.522084 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:41:37.524544 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.524090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:37.524742 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.524612 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:41:37.524742 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.524669 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:41:37.524742 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.524710 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:41:37.524742 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.524717 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:41:37.526154 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.526141 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:41:37.526220 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:37.526207 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:41:39.652991 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.652973 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:41:39.653129 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.653116 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:41:39.653513 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.653237 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:41:39.653513 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.653332 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:41:39.658643 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.657710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.658643 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.658101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.658643 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.658306 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:39.659746 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.659724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.659843 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.659796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:39.663754 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.663737 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6fqf" Oct 29 21:41:39.676505 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.676488 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6fqf" Oct 29 21:41:39.707192 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.707171 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:41:39.717677 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:39.717754 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.717813 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.717813 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46k8\" (UniqueName: \"kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.717915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.717915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.717915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.717915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.717915 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47szc\" (UniqueName: \"kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.717990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.718126 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtffq\" (UniqueName: \"kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.718514 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtv9\" (UniqueName: \"kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8w4\" (UniqueName: \"kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.719052 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnw6x\" (UniqueName: \"kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.718998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zp86\" (UniqueName: \"kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.719702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgncj\" (UniqueName: \"kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5m4\" (UniqueName: \"kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.719986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.720046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.720102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.720495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.720135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.721034 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.720194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.746745 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.746727 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:41:39.746849 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.746743 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:41:39.746937 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.746921 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:41:39.747012 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.746999 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:41:39.747064 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747054 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:39.747197 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747183 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:41:39.747339 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747322 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:41:39.747405 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747353 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:39.747405 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747367 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:41:39.747509 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747424 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:39.747560 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747523 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:41:39.747560 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747550 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:41:39.747654 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747581 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:41:39.747654 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747598 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:41:39.747654 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747640 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:39.747765 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747684 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:41:39.747765 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747687 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:41:39.747765 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747713 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:41:39.747765 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747640 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:41:39.747929 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.747775 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:41:39.820614 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.820614 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.820614 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtffq\" (UniqueName: \"kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.820842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.820997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtv9\" (UniqueName: \"kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821197 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821249 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821288 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821383 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8w4\" (UniqueName: \"kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821434 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnw6x\" (UniqueName: \"kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821565 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.821842 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zp86\" (UniqueName: \"kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821720 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821761 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821808 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgncj\" (UniqueName: \"kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.822598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822097 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5m4\" (UniqueName: \"kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822719 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822776 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822791 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822848 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822882 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822903 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.823441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.822997 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.824106 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.823377 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.824106 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.823443 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.824106 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.823503 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.824106 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.823518 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.824106 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.823541 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.824292 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824133 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.824292 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824196 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.824292 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824233 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.824399 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824308 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.824399 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.821683 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.824399 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824354 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.824399 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.824387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.824537 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.824453 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:39.825047 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825021 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.825159 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825097 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825159 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.825159 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825141 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825203 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825220 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825259 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825267 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825320 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825302 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.825598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825327 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.825598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825386 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.825598 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825499 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.825741 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825618 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.825741 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825741 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825741 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825741 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.825865 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:41:40.325844513 +0000 UTC m=+3.100071613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825877 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.825965 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.825991 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826067 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826139 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.826254 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826274 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826233 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826308 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.826300 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:40.326285085 +0000 UTC m=+3.100512210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46k8\" (UniqueName: \"kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47szc\" (UniqueName: \"kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826572 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826599 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826691 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:39.826857 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826832 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.826900 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827035 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827440 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827522 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827577 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827637 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827713 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.828348 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.827895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.830976 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.830898 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.849768 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.849743 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtffq\" (UniqueName: \"kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:39.850219 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.850198 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zp86\" (UniqueName: \"kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:39.850659 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.850630 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtv9\" (UniqueName: \"kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.852732 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.852701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8w4\" (UniqueName: \"kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.854210 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.854188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgncj\" (UniqueName: \"kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.855288 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.855268 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47szc\" (UniqueName: \"kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:39.859521 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.859502 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46k8\" (UniqueName: \"kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:39.859822 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.859731 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:39.859822 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.859760 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:39.859822 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.859774 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:39.859991 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:39.859834 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:41:40.359821352 +0000 UTC m=+3.134048469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:39.861992 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.861967 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5m4\" (UniqueName: \"kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:41:39.863209 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.863166 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnw6x\" (UniqueName: \"kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.945898 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.945867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:39.953001 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.952823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2k6br" Oct 29 21:41:39.957142 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:39.957113 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0a27b_37b7_4dc0_932d_913d95a9d0e4.slice/crio-6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04 WatchSource:0}: Error finding container 6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04: Status 404 returned error can't find the container with id 6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04 Oct 29 21:41:39.963716 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:39.963677 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7672c73e_69be_484f_84ae_12c26a104e59.slice/crio-d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae WatchSource:0}: Error finding container d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae: Status 404 returned error can't find the container with id d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae Oct 29 21:41:39.969087 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.969040 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:41:39.980544 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:39.980516 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad6f194_bef0_48b9_bf0a_725194f1703a.slice/crio-618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f WatchSource:0}: Error finding container 618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f: Status 404 returned error can't find the container with id 618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f Oct 29 21:41:39.988007 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.987983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:41:39.995583 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.995530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:41:39.997686 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:39.997666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpkwp" Oct 29 21:41:40.000194 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:40.000169 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427f41b3_dae2_47cc_a0f1_749c57377d7e.slice/crio-1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9 WatchSource:0}: Error finding container 1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9: Status 404 returned error can't find the container with id 1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9 Oct 29 21:41:40.005831 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:40.005805 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2889a3a6_52aa_45d9_a13c_d4fb5cf6abba.slice/crio-1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd WatchSource:0}: Error finding container 1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd: Status 404 returned error can't find the container with id 1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd Oct 29 21:41:40.018733 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.018695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:41:40.021063 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.021041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:41:40.024511 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.024491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:40.030089 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:40.030054 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3042dc3_787e_46c3_80b8_dfb9294328d7.slice/crio-59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155 WatchSource:0}: Error finding container 59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155: Status 404 returned error can't find the container with id 59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155 Oct 29 21:41:40.033706 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:40.033682 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7aa62f_dc8e_4339_bfa0_7553135f02c7.slice/crio-ad2d44bac04ac06d0125d82a7202d30b1c2f693b05b5296b3b092bb1564715ed WatchSource:0}: Error finding container ad2d44bac04ac06d0125d82a7202d30b1c2f693b05b5296b3b092bb1564715ed: Status 404 returned error can't find the container with id ad2d44bac04ac06d0125d82a7202d30b1c2f693b05b5296b3b092bb1564715ed Oct 29 21:41:40.039589 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:41:40.039119 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340734d7_e5e9_440d_8d36_d88ab0c4bf83.slice/crio-36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce WatchSource:0}: Error finding container 36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce: Status 404 returned error can't find the container with id 36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce Oct 29 21:41:40.096706 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.096655 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:40.329412 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.329136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:40.329412 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.329280 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:40.329412 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.329363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:40.329602 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.329420 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:41:41.329402542 +0000 UTC m=+4.103629644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:40.329602 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.329474 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:40.329602 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.329529 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:41.329513246 +0000 UTC m=+4.103740365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:40.430809 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.430163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:40.430809 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.430329 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:40.430809 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.430347 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:40.430809 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.430359 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:40.430809 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:40.430410 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:41:41.430393479 +0000 UTC m=+4.204620597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:40.679308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.679204 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:39 +0000 UTC" deadline="2028-06-30 02:11:03.498197729 +0000 UTC" Oct 29 21:41:40.679308 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.679260 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14620h29m22.818940993s" Oct 29 21:41:40.775937 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.775158 2574 generic.go:356] "Generic (PLEG): container finished" podID="186cd4ee51f579a3597049aa602c4c28" containerID="9153a8c88cb7d47d0af11b3ddb6cd5488470a7b4de7e0453f6b917f3566680eb" exitCode=0 Oct 29 21:41:40.775937 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.775506 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerDied","Data":"9153a8c88cb7d47d0af11b3ddb6cd5488470a7b4de7e0453f6b917f3566680eb"} Oct 29 21:41:40.781907 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.781505 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" event={"ID":"2787fae5ca7f0b39f010401de9cba34b","Type":"ContainerStarted","Data":"64702d90fc08be7e1a6fdddfdeb073bbb7c3526bb3e1468df031a6b8a0ce8d07"} Oct 29 21:41:40.791893 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.791835 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58pn4" event={"ID":"9f7aa62f-dc8e-4339-bfa0-7553135f02c7","Type":"ContainerStarted","Data":"ad2d44bac04ac06d0125d82a7202d30b1c2f693b05b5296b3b092bb1564715ed"} Oct 29 21:41:40.806268 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.806169 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpkwp" event={"ID":"5921e77d-c1b2-448a-bd09-fc72e935dfc7","Type":"ContainerStarted","Data":"6fa5427f65461829783d2c970a08af912655ce5bfed29b540466f4b5a39ee3e8"} Oct 29 21:41:40.832145 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.832118 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2k6br" event={"ID":"7672c73e-69be-484f-84ae-12c26a104e59","Type":"ContainerStarted","Data":"d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae"} Oct 29 21:41:40.835258 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.835227 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce"} Oct 29 21:41:40.838861 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.838829 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8j8nv" event={"ID":"a3042dc3-787e-46c3-80b8-dfb9294328d7","Type":"ContainerStarted","Data":"59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155"} Oct 29 21:41:40.846347 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.846323 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" event={"ID":"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba","Type":"ContainerStarted","Data":"1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd"} Oct 29 21:41:40.853728 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.853681 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerStarted","Data":"1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9"} Oct 29 21:41:40.858093 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.858054 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkxjk" event={"ID":"fad6f194-bef0-48b9-bf0a-725194f1703a","Type":"ContainerStarted","Data":"618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f"} Oct 29 21:41:40.869241 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:40.869219 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znvhf" event={"ID":"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4","Type":"ContainerStarted","Data":"6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04"} Oct 29 21:41:41.337419 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.337385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:41.337626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.337470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:41.337626 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.337620 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:41.337780 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.337688 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.337668876 +0000 UTC m=+6.111895982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:41.338155 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.338133 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:41.338247 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.338215 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.338181971 +0000 UTC m=+6.112409078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:41.438317 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.438287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:41.438447 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.438437 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:41.438502 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.438455 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:41.438502 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.438468 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:41.438668 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.438516 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.438499513 +0000 UTC m=+6.212726633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:41.682617 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.682518 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:39 +0000 UTC" deadline="2028-03-28 15:52:22.513062474 +0000 UTC" Oct 29 21:41:41.682617 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.682571 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12378h10m40.830495059s" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.757419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.757543 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.757917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.758024 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.758109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:41.758297 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:41.758179 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:41.890227 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.890150 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerStarted","Data":"4405e53be90654dd9618dd76fb755e27802f66985487f32fdb9128c52e46d452"} Oct 29 21:41:41.928156 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:41.928095 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" podStartSLOduration=3.928059444 podStartE2EDuration="3.928059444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:40.853694056 +0000 UTC m=+3.627921197" watchObservedRunningTime="2026-10-29 21:41:41.928059444 +0000 UTC m=+4.702286568" Oct 29 21:41:42.897023 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:42.896991 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="2d3c1e51497063011619f9ded17a3958ebaf3599ad45b63ebb3e740907d66be1" exitCode=0 Oct 29 21:41:42.897457 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:42.897212 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"2d3c1e51497063011619f9ded17a3958ebaf3599ad45b63ebb3e740907d66be1"} Oct 29 21:41:42.963814 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:42.963159 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" podStartSLOduration=4.963141084 podStartE2EDuration="4.963141084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:41.928692216 +0000 UTC m=+4.702919340" watchObservedRunningTime="2026-10-29 21:41:42.963141084 +0000 UTC m=+5.737368219" Oct 29 21:41:43.353744 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.353712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:43.353920 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.353776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:43.353920 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.353902 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.354130 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.353956 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.353939584 +0000 UTC m=+10.128166700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.354376 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.354352 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.354472 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.354405 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.354389706 +0000 UTC m=+10.128616822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.454367 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.454326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:43.454536 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.454477 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:43.454536 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.454496 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:43.454536 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.454507 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.454681 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.454559 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:41:47.454540382 +0000 UTC m=+10.228767484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.753068 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.753043 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:43.753202 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.753183 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:43.758561 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.753637 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:43.758561 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.753779 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:43.758561 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:43.756219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:43.758561 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:43.756402 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:44.909761 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:44.909677 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkxjk" event={"ID":"fad6f194-bef0-48b9-bf0a-725194f1703a","Type":"ContainerStarted","Data":"f06061d4e46cefe034a878d7f586f297c759a808bc9e7d9415f7a3f5304cc2e4"} Oct 29 21:41:44.914607 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:44.914577 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znvhf" event={"ID":"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4","Type":"ContainerStarted","Data":"66ffabd2f724bc041fcd49b00a77ef2913594b6f0e17279b411c79849d7acdcd"} Oct 29 21:41:44.947969 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:44.947917 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wkxjk" podStartSLOduration=3.40116467 podStartE2EDuration="7.947901344s" totalImagesPullingTime="4.546736674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.947322406 +0000 UTC m=+7.721549529" watchObservedRunningTime="2026-10-29 21:41:44.947901344 +0000 UTC m=+7.722128468" Oct 29 21:41:45.602592 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.602560 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:45.605118 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.603135 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:41:45.640865 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.640814 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-znvhf" podStartSLOduration=4.863424945 podStartE2EDuration="8.640797805s" totalImagesPullingTime="3.77737286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.987132662 +0000 UTC m=+7.761359803" watchObservedRunningTime="2026-10-29 21:41:45.640797805 +0000 UTC m=+8.415024929" Oct 29 21:41:45.752506 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.752467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:45.752623 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:45.752602 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:45.752681 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.752668 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:45.752740 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:45.752724 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:45.752790 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.752763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:45.752840 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:45.752805 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:45.919042 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:45.918971 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" event={"ID":"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba","Type":"ContainerStarted","Data":"f698d3470b92499baf84684dd4641f08ad760c7881d5098c0eca8cb18ac7653a"} Oct 29 21:41:46.921044 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:46.921012 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:41:47.219932 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.219912 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:41:47.383573 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.383543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:47.383674 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.383608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:47.383713 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.383693 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:47.383741 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.383732 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:41:55.383720098 +0000 UTC m=+18.157947199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:47.383778 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.383774 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:47.383861 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.383794 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:55.383787181 +0000 UTC m=+18.158014282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:47.484895 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.484804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:47.485115 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.485061 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:47.485115 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.485112 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:47.485230 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.485127 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:47.485230 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.485197 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:41:55.485177807 +0000 UTC m=+18.259404980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:47.694237 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.694140 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:41:47.219929576Z","UUID":"477f8d44-fe89-46d2-923c-df4667f82683","Handler":null,"Name":"","Endpoint":""} Oct 29 21:41:47.696785 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.696762 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:41:47.696921 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.696793 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.752574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.752725 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.753095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.753234 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.753294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:47.755044 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:47.753360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:47.925813 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.925593 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" event={"ID":"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba","Type":"ContainerStarted","Data":"e478b3e6e5dafb9cba7a11768b25e0e68e7745a1a068481aa7bbe37d0c142797"} Oct 29 21:41:47.927480 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.927458 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="667919cfae05369d7db5cdd56b168c2b9c4960eda857a0cf0ed49852d5530ed6" exitCode=0 Oct 29 21:41:47.927589 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.927526 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"667919cfae05369d7db5cdd56b168c2b9c4960eda857a0cf0ed49852d5530ed6"} Oct 29 21:41:47.930902 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.930878 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpkwp" event={"ID":"5921e77d-c1b2-448a-bd09-fc72e935dfc7","Type":"ContainerStarted","Data":"b908411e2bd8cf11b1c96925de55078e1858a4356b3632b0bfcd4cd379cb1257"} Oct 29 21:41:47.933185 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:47.932794 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2k6br" event={"ID":"7672c73e-69be-484f-84ae-12c26a104e59","Type":"ContainerStarted","Data":"e8fd7e589d92e558e6e02930128e6c497fbc60945b9c592ae1a2faca5265cd07"} Oct 29 21:41:48.017756 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.017668 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2k6br" podStartSLOduration=3.27264282 podStartE2EDuration="11.017651399s" totalImagesPullingTime="7.745008579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:48.017456883 +0000 UTC m=+10.791684007" watchObservedRunningTime="2026-10-29 21:41:48.017651399 +0000 UTC m=+10.791878508" Oct 29 21:41:48.057872 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.057813 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wpkwp" podStartSLOduration=3.722553622 podStartE2EDuration="11.057796387s" totalImagesPullingTime="7.335242765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:48.057315442 +0000 UTC m=+10.831542592" watchObservedRunningTime="2026-10-29 21:41:48.057796387 +0000 UTC m=+10.832023512" Oct 29 21:41:48.937133 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.937097 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" event={"ID":"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba","Type":"ContainerStarted","Data":"612ef782a788adcd1fde75201a3e68bf00e79e212f0c77bcd767df183a773b59"} Oct 29 21:41:48.939575 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.939540 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="b4fc24628bfe0172ada8e1c55498e7afd1079668d22359747e83b6e17abc51ab" exitCode=0 Oct 29 21:41:48.939685 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.939634 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"b4fc24628bfe0172ada8e1c55498e7afd1079668d22359747e83b6e17abc51ab"} Oct 29 21:41:48.977646 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:48.977595 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" podStartSLOduration=4.113764616 podStartE2EDuration="11.97757776s" totalImagesPullingTime="7.863813144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:48.976924294 +0000 UTC m=+11.751151442" watchObservedRunningTime="2026-10-29 21:41:48.97757776 +0000 UTC m=+11.751804885" Oct 29 21:41:49.749899 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.749868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:49.749899 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.749890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:49.750027 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.749882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:49.750027 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:49.750000 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:49.750156 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:49.750051 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:49.750223 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:49.750199 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:49.943199 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.943108 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58pn4" event={"ID":"9f7aa62f-dc8e-4339-bfa0-7553135f02c7","Type":"ContainerStarted","Data":"4d770e5729ffc64ee41ff64c6f49672f741b29f3145356eb408f895d913127b3"} Oct 29 21:41:49.945485 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.945461 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8j8nv" event={"ID":"a3042dc3-787e-46c3-80b8-dfb9294328d7","Type":"ContainerStarted","Data":"3c4cb40748767fb66385350ffc64d698ae76eca3624fccb24275abc471e12482"} Oct 29 21:41:49.948554 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.948522 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="23a0f69ea8e3ae84b49ed1f47b384c38fc14b3f919d3e636d7cbeeeedfed137e" exitCode=0 Oct 29 21:41:49.948661 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.948589 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"23a0f69ea8e3ae84b49ed1f47b384c38fc14b3f919d3e636d7cbeeeedfed137e"} Oct 29 21:41:49.984365 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:49.984313 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-58pn4" podStartSLOduration=3.378512018 podStartE2EDuration="12.984297095s" totalImagesPullingTime="9.605785077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:49.983320056 +0000 UTC m=+12.757547180" watchObservedRunningTime="2026-10-29 21:41:49.984297095 +0000 UTC m=+12.758524219" Oct 29 21:41:50.063535 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:50.063495 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8j8nv" podStartSLOduration=5.603789526 podStartE2EDuration="13.063481941s" totalImagesPullingTime="7.459692415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:50.063244962 +0000 UTC m=+12.837472087" watchObservedRunningTime="2026-10-29 21:41:50.063481941 +0000 UTC m=+12.837709066" Oct 29 21:41:51.750064 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:51.750034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:51.750676 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:51.750183 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:51.750676 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:51.750185 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:51.750676 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:51.750257 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:51.750676 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:51.750307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:51.750676 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:51.750378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:51.956154 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:51.955852 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="0180a39e5637c6737c2b82fdf69889686f6d768f0c6a8467d1ad661c4462c3f5" exitCode=0 Oct 29 21:41:51.956154 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:51.955942 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"0180a39e5637c6737c2b82fdf69889686f6d768f0c6a8467d1ad661c4462c3f5"} Oct 29 21:41:52.961512 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:52.961474 2574 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="e187ae09f1d2f930111a664e1459898a6692d6526b6452ef10811eb65ef27035" exitCode=0 Oct 29 21:41:52.961956 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:52.961523 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerDied","Data":"e187ae09f1d2f930111a664e1459898a6692d6526b6452ef10811eb65ef27035"} Oct 29 21:41:53.752901 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:53.752863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:53.753037 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:53.752962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:53.753037 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:53.752987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:53.753136 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:53.753052 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:53.753136 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:53.753125 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:53.753198 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:53.753179 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:53.966370 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:53.966345 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-htxp7" event={"ID":"427f41b3-dae2-47cc-a0f1-749c57377d7e","Type":"ContainerStarted","Data":"c0f6f0e569dc7fc92279a1509cf8c98b1eb2df7210af612d5128b81c627f3573"} Oct 29 21:41:54.031871 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.031802 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-htxp7" podStartSLOduration=7.389128424 podStartE2EDuration="17.031791369s" totalImagesPullingTime="9.642662945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:54.020669876 +0000 UTC m=+16.794897025" watchObservedRunningTime="2026-10-29 21:41:54.031791369 +0000 UTC m=+16.806018491" Oct 29 21:41:54.971446 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971412 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"7b0aadfe7cf550433d02beb9c3aff34f832f3f3c20c2a36d4f6208283930fe66"} Oct 29 21:41:54.971782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971511 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"328591698baa2d5bf1a34954d27ab978f285a6bb3ea63845df96cf92e9a56876"} Oct 29 21:41:54.971782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971543 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"b570071373179dc09a28532dbf63d5a61cabccf1dfebf569d611482f873d75ac"} Oct 29 21:41:54.971782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971578 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"4cac81a7d7d2323e96daf3aa822c169a38eb447e1e2d5fbf334a20754b147c06"} Oct 29 21:41:54.971782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971608 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"ddfd3896ddf4c0ee082c71fb0a407ccfc10c5c4d33efdaa105f1c380e1bd08bf"} Oct 29 21:41:54.971782 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:54.971643 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"fd7824134c905cbe4b50316134ce361736406455446b84b185dc0deb7fd21093"} Oct 29 21:41:55.440280 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.440246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:55.440416 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.440296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:55.440416 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.440397 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:55.440416 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.440397 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:55.440521 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.440460 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret podName:29210f25-02b2-48e3-9d5b-24fbf755276a nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.440444548 +0000 UTC m=+34.214671662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret") pod "global-pull-secret-syncer-2tdpb" (UID: "29210f25-02b2-48e3-9d5b-24fbf755276a") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:55.440521 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.440475 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.440467774 +0000 UTC m=+34.214694875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:55.541350 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.541321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:55.541450 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.541437 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:55.541497 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.541452 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:55.541497 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.541463 2574 projected.go:196] Error preparing data for projected volume kube-api-access-l6rln for pod openshift-network-diagnostics/network-check-target-2thvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:55.541551 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.541505 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln podName:97d32947-bedc-47aa-bd2e-48a33b4a309f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.541492926 +0000 UTC m=+34.315720027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6rln" (UniqueName: "kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln") pod "network-check-target-2thvl" (UID: "97d32947-bedc-47aa-bd2e-48a33b4a309f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:55.749728 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.749671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:55.749728 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.749699 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:55.749728 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:55.749705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:55.749925 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.749797 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:55.749925 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.749863 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:55.750022 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:55.749938 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:56.978573 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:56.978544 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"7e714f4820533675a9a5ade349491c43bb5fd269046692e8699caede12cbfdcf"} Oct 29 21:41:57.752719 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:57.752692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:57.752840 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:57.752794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:57.752840 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:57.752803 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:57.752840 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:57.752823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:57.752958 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:57.752895 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:57.752989 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:57.752974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:41:58.986341 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:58.986307 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" event={"ID":"340734d7-e5e9-440d-8d36-d88ab0c4bf83","Type":"ContainerStarted","Data":"d22722155615ad902bbb93211f8cd5b9e09173175b092052440ca7b2a6c99311"} Oct 29 21:41:58.986944 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:58.986664 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:58.986944 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:58.986743 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:58.986944 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:58.986790 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:59.003034 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.003009 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:59.003548 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.003530 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:41:59.128511 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.128464 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" podStartSLOduration=7.689870863 podStartE2EDuration="22.128450872s" totalImagesPullingTime="14.438580009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:59.071782203 +0000 UTC m=+21.846009327" watchObservedRunningTime="2026-10-29 21:41:59.128450872 +0000 UTC m=+21.902677995" Oct 29 21:41:59.752718 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.752691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:41:59.752838 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.752775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:41:59.752838 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:59.752781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:41:59.752838 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:41:59.752798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:41:59.752932 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:59.752864 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:41:59.752975 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:41:59.752944 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:42:01.070602 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.070380 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tdpb"] Oct 29 21:42:01.070906 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.070651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:01.070906 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:01.070733 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:42:01.080841 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.080811 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2thvl"] Oct 29 21:42:01.081015 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.080914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:01.081015 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:01.080986 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:42:01.081474 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.081450 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jvnws"] Oct 29 21:42:01.081603 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:01.081559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:01.081683 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:01.081662 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:42:02.749967 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:02.749936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:02.750452 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:02.749942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:02.750452 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:02.750041 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:42:02.750452 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:02.749942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:02.750452 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:02.750146 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:42:02.750452 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:02.750227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:42:04.749952 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:04.749914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:04.750458 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:04.749927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:04.750458 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:04.750019 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:42:04.750458 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:04.750125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:04.750458 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:04.750134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:42:04.750458 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:04.750218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:42:06.123173 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.123139 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:06.123626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.123261 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:42:06.123924 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.123904 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:06.750040 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.750013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:06.750194 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.750052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:06.750194 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:06.750019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:06.750194 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:06.750147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tdpb" podUID="29210f25-02b2-48e3-9d5b-24fbf755276a" Oct 29 21:42:06.750336 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:06.750286 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jvnws" podUID="16564c0f-cddb-477c-95e4-e9d3a9996bd2" Oct 29 21:42:06.750399 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:06.750379 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2thvl" podUID="97d32947-bedc-47aa-bd2e-48a33b4a309f" Oct 29 21:42:07.028798 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.028778 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeReady" Oct 29 21:42:07.028920 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.028886 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 21:42:07.094233 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.094209 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hzqgn"] Oct 29 21:42:07.096998 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.096983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.100671 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.100647 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p75bp"] Oct 29 21:42:07.103243 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.103221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.113203 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.113185 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:07.113284 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.113236 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:07.113284 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.113274 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:07.113351 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.113263 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:07.118064 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.118044 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzqgn"] Oct 29 21:42:07.148495 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.148470 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p75bp"] Oct 29 21:42:07.213379 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.213358 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:07.213468 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.213406 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:07.213468 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.213410 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:07.225605 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkm4f\" (UniqueName: \"kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.225656 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.225656 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.225715 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.225784 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.225819 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.225795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drz2\" (UniqueName: \"kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.326935 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.326897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.326935 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.326924 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drz2\" (UniqueName: \"kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.327011 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.326956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkm4f\" (UniqueName: \"kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.327042 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.327017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.327097 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.327047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.327182 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:07.327161 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:07.327242 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:07.327234 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.827213742 +0000 UTC m=+30.601440843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:07.327296 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.327269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.327547 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.327530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.327588 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.327547 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.330688 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.330672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.350463 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.350443 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drz2\" (UniqueName: \"kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.350726 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.350709 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkm4f\" (UniqueName: \"kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.408644 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.408626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:07.712287 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.712230 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzqgn"] Oct 29 21:42:07.726972 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:07.715406 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d4cae5_2c19_4166_9d60_4fcc17b689b6.slice/crio-6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d WatchSource:0}: Error finding container 6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d: Status 404 returned error can't find the container with id 6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d Oct 29 21:42:07.830956 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:07.830927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:07.831119 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:07.831100 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:07.831193 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:07.831168 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:08.831149042 +0000 UTC m=+31.605376160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:08.004904 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.004845 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzqgn" event={"ID":"38d4cae5-2c19-4166-9d60-4fcc17b689b6","Type":"ContainerStarted","Data":"6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d"} Oct 29 21:42:08.749945 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.749915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:08.750496 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.749971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:08.750496 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.749983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:08.765214 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.765192 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:08.765316 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.765193 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:08.765955 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.765937 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:08.776210 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.776195 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:08.776369 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.776352 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:08.776467 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.776399 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:08.776467 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.776430 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:08.838030 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:08.838005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:08.838195 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:08.838174 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:08.838261 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:08.838236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:10.83822024 +0000 UTC m=+33.612447345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:10.854219 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.854183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:10.854541 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:10.854331 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:10.854541 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:10.854399 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:14.854383924 +0000 UTC m=+37.628611026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:10.880016 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.879985 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jpxgr"] Oct 29 21:42:10.927506 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.927467 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpxgr"] Oct 29 21:42:10.927626 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.927605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:10.931711 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.931687 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:42:10.953416 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.953400 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:10.955270 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.955251 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:42:10.994576 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.994532 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:10.994576 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.994549 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:10.994708 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.994691 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:10.994750 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:10.994740 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 21:42:11.012859 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.012841 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzqgn" event={"ID":"38d4cae5-2c19-4166-9d60-4fcc17b689b6","Type":"ContainerStarted","Data":"d6035819ba4c1b2d071a59577453ecf9bffb46cfc1bf0e719e7b04f70b3f5769"} Oct 29 21:42:11.016527 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.016508 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:11.016595 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.016532 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:11.016595 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.016542 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 21:42:11.016595 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.016566 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sjp55\"" Oct 29 21:42:11.055686 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.055764 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.055764 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.055764 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.055925 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgc7\" (UniqueName: \"kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.055966 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.055940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.056056 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.056039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.056127 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.056091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.056162 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.056143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.058417 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.058382 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hzqgn" podStartSLOduration=1.450337973 podStartE2EDuration="4.05837064s" totalImagesPullingTime="2.608032667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:11.058068869 +0000 UTC m=+33.832295993" watchObservedRunningTime="2026-10-29 21:42:11.05837064 +0000 UTC m=+33.832597763" Oct 29 21:42:11.156657 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.156781 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.156781 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.156781 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.156938 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.156807 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.156938 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgc7\" (UniqueName: \"kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.156938 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.156858 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.65684283 +0000 UTC m=+34.431069931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.156938 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.156938 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.157189 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.157189 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.156987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.157189 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.157011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.157189 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.157117 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:11.157189 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.157186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.157422 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.157241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.657205547 +0000 UTC m=+34.431432651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : configmap "kube-rbac-proxy" not found Oct 29 21:42:11.157422 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.157284 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.157614 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.157591 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.157902 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.157887 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.184549 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.184525 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.184989 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.184967 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgc7\" (UniqueName: \"kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.301445 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.301399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:11.459441 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.459412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:11.459551 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.459471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:11.459597 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.459560 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:11.459633 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.459623 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:43.459605034 +0000 UTC m=+66.233832153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : secret "metrics-daemon-secret" not found Oct 29 21:42:11.461643 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.461624 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:11.508259 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.508236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:11.542696 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.542669 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r9mdg"] Oct 29 21:42:11.560751 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.560705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:11.563096 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.563057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:11.567316 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.567295 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.568050 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.567977 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r9mdg"] Oct 29 21:42:11.583204 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.583182 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 21:42:11.584050 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.584023 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 21:42:11.584172 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.584143 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 21:42:11.584366 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.584239 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 21:42:11.584366 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.584355 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c7spd\"" Oct 29 21:42:11.641702 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.641676 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:42:11.644459 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.644435 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh"] Oct 29 21:42:11.649236 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:11.649213 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b49a2a6_8054_4068_bd32_fdd07397691a.slice/crio-f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e WatchSource:0}: Error finding container f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e: Status 404 returned error can't find the container with id f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e Oct 29 21:42:11.661178 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.661151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.661178 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.661175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8wq\" (UniqueName: \"kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.661300 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.661211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.661335 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.661306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.661335 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.661309 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:11.661391 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.661363 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.661348265 +0000 UTC m=+35.435575379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : configmap "kube-rbac-proxy" not found Oct 29 21:42:11.661424 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.661399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:11.661520 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.661507 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.661561 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:11.661551 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.661540442 +0000 UTC m=+35.435767543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.669547 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.669526 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh"] Oct 29 21:42:11.669640 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.669611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:11.684242 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.684224 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:11.684242 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.684233 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxdl2\"" Oct 29 21:42:11.684355 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.684225 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 21:42:11.761774 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.761755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.761852 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.761802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.761852 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.761821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8wq\" (UniqueName: \"kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.761947 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.761852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4rr\" (UniqueName: \"kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr\") pod \"migrator-66df5d496b-f9wjh\" (UID: \"507ad5fa-55a9-4719-aa6a-12a777a05c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:11.762384 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.762369 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.764069 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.764052 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.784823 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.784795 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8wq\" (UniqueName: \"kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.794647 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.794630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:11.830468 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.830447 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tdpb"] Oct 29 21:42:11.833974 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:11.833953 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29210f25_02b2_48e3_9d5b_24fbf755276a.slice/crio-40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe WatchSource:0}: Error finding container 40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe: Status 404 returned error can't find the container with id 40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe Oct 29 21:42:11.865396 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.865366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4rr\" (UniqueName: \"kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr\") pod \"migrator-66df5d496b-f9wjh\" (UID: \"507ad5fa-55a9-4719-aa6a-12a777a05c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:11.894185 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.894158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:11.896036 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.896015 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4rr\" (UniqueName: \"kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr\") pod \"migrator-66df5d496b-f9wjh\" (UID: \"507ad5fa-55a9-4719-aa6a-12a777a05c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:11.994625 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:11.994404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:12.016650 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.016624 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tdpb" event={"ID":"29210f25-02b2-48e3-9d5b-24fbf755276a","Type":"ContainerStarted","Data":"40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe"} Oct 29 21:42:12.017861 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.017842 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" event={"ID":"0b49a2a6-8054-4068-bd32-fdd07397691a","Type":"ContainerStarted","Data":"f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e"} Oct 29 21:42:12.105129 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.105051 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2thvl"] Oct 29 21:42:12.107885 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:12.107860 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d32947_bedc_47aa_bd2e_48a33b4a309f.slice/crio-41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1 WatchSource:0}: Error finding container 41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1: Status 404 returned error can't find the container with id 41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1 Oct 29 21:42:12.225017 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.224991 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r9mdg"] Oct 29 21:42:12.228307 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:12.228284 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2acd49a_da89_47ed_b1cb_40e068e16d85.slice/crio-f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0 WatchSource:0}: Error finding container f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0: Status 404 returned error can't find the container with id f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0 Oct 29 21:42:12.296105 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.296059 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh"] Oct 29 21:42:12.299911 ip-10-0-128-100 kubenswrapper[2574]: W1029 21:42:12.299884 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507ad5fa_55a9_4719_aa6a_12a777a05c5e.slice/crio-d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672 WatchSource:0}: Error finding container d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672: Status 404 returned error can't find the container with id d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672 Oct 29 21:42:12.671625 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.671592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:12.671796 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:12.671677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:12.671866 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:12.671799 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.671866 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:12.671860 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:14.67184052 +0000 UTC m=+37.446067638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.672280 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:12.672260 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:12.672370 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:12.672317 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:14.672301611 +0000 UTC m=+37.446528719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : configmap "kube-rbac-proxy" not found Oct 29 21:42:13.024332 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:13.024194 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" event={"ID":"e2acd49a-da89-47ed-b1cb-40e068e16d85","Type":"ContainerStarted","Data":"f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0"} Oct 29 21:42:13.026173 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:13.026137 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" event={"ID":"507ad5fa-55a9-4719-aa6a-12a777a05c5e","Type":"ContainerStarted","Data":"d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672"} Oct 29 21:42:13.027958 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:13.027927 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2thvl" event={"ID":"97d32947-bedc-47aa-bd2e-48a33b4a309f","Type":"ContainerStarted","Data":"41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1"} Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:14.688460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:14.688557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.688744 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.688803 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:18.688786564 +0000 UTC m=+41.463013675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.688845 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:14.689219 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.688877 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:18.688869953 +0000 UTC m=+41.463097056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : configmap "kube-rbac-proxy" not found Oct 29 21:42:14.890132 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:14.890101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:14.890298 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.890248 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:14.890348 ip-10-0-128-100 kubenswrapper[2574]: E1029 21:42:14.890319 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:22.890299438 +0000 UTC m=+45.664526562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:15.036694 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:15.036035 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" event={"ID":"507ad5fa-55a9-4719-aa6a-12a777a05c5e","Type":"ContainerStarted","Data":"80095be0be74826c27d2f2d9e45217ee168467719aacfc48165b35841e81e4e3"} Oct 29 21:42:15.077962 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:15.077898 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" podStartSLOduration=2.086514569 podStartE2EDuration="4.077879912s" totalImagesPullingTime="1.991365343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:15.076020773 +0000 UTC m=+37.850247898" watchObservedRunningTime="2026-10-29 21:42:15.077879912 +0000 UTC m=+37.852107035" Oct 29 21:42:16.040967 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:16.040416 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" event={"ID":"0b49a2a6-8054-4068-bd32-fdd07397691a","Type":"ContainerStarted","Data":"eeeccef5255bcb381a40028ec250779bcd4021a38e07f03142335a0b5c792b34"} Oct 29 21:42:16.044304 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:16.044274 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" event={"ID":"e2acd49a-da89-47ed-b1cb-40e068e16d85","Type":"ContainerStarted","Data":"edcf3bb5fb74cf7d739f60a6653c0027610fbe4c6d4aecd5c6352114999339e7"} Oct 29 21:42:16.080418 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:16.080367 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" podStartSLOduration=2.755910202 podStartE2EDuration="6.080349486s" totalImagesPullingTime="3.324439284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:16.079679077 +0000 UTC m=+38.853906201" watchObservedRunningTime="2026-10-29 21:42:16.080349486 +0000 UTC m=+38.854576611" Oct 29 21:42:17.049397 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:17.049316 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2thvl" event={"ID":"97d32947-bedc-47aa-bd2e-48a33b4a309f","Type":"ContainerStarted","Data":"372ea263176edc7bdc8e90fa034bbbb8feb5b19ed2fbfef08ecc6d4fd05ded1e"} Oct 29 21:42:17.049397 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:17.049380 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:17.108329 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:17.108271 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" podStartSLOduration=2.766732429 podStartE2EDuration="6.10825338s" totalImagesPullingTime="3.341520951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:16.124639595 +0000 UTC m=+38.898866717" watchObservedRunningTime="2026-10-29 21:42:17.10825338 +0000 UTC m=+39.882480504" Oct 29 21:42:18.053029 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:18.052989 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tdpb" event={"ID":"29210f25-02b2-48e3-9d5b-24fbf755276a","Type":"ContainerStarted","Data":"51552a26a614337c034a54b7e6fb5468feeb2c1aae0b4258008f5b7537db7ef1"} Oct 29 21:42:18.091176 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:18.091108 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2thvl" podStartSLOduration=36.596354389 podStartE2EDuration="41.091059952s" totalImagesPullingTime="4.494705563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:17.107668351 +0000 UTC m=+39.881895500" watchObservedRunningTime="2026-10-29 21:42:18.091059952 +0000 UTC m=+40.865287080" Oct 29 21:42:18.091374 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:18.091346 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2tdpb" podStartSLOduration=35.058489642 podStartE2EDuration="41.091340179s" totalImagesPullingTime="6.032850537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:18.090679755 +0000 UTC m=+40.864906889" watchObservedRunningTime="2026-10-29 21:42:18.091340179 +0000 UTC m=+40.865567303" Oct 29 21:42:18.174337 ip-10-0-128-100 kubenswrapper[2574]: I1029 21:42:18.174250 2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:18.174423 ip-10-0-128-100 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 21:42:18.189475 ip-10-0-128-100 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 21:42:18.189640 ip-10-0-128-100 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 21:42:18.189686 ip-10-0-128-100 systemd[1]: kubelet.service: Consumed 2.286s CPU time. Oct 29 21:42:18.203713 ip-10-0-128-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:42:18.298052 ip-10-0-128-100 kubenswrapper[4543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:18.298052 ip-10-0-128-100 kubenswrapper[4543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:42:18.298052 ip-10-0-128-100 kubenswrapper[4543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:18.298052 ip-10-0-128-100 kubenswrapper[4543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:18.300111 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300026 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.300111 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300110 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300114 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300117 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300120 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300125 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300129 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300133 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300136 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300139 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300142 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300145 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300147 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300150 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300152 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300155 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300158 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300160 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300163 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300165 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300168 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.300183 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300171 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300174 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300176 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300179 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300181 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300184 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300195 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300198 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300201 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300203 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300206 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300208 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300211 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300213 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300216 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300218 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300221 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300223 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300226 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.300693 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300229 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300233 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300236 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300239 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300242 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300244 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300246 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300249 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300251 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300254 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300256 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300260 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300263 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300265 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300267 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300270 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300273 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300276 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300279 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.301200 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300281 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300284 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300286 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300289 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300291 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300294 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300296 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300299 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300301 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300304 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300307 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300309 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300312 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300314 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300317 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300319 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300322 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300324 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300327 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.301675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300329 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300332 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300334 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300336 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300339 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300342 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300345 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300347 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300350 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300353 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300356 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300359 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300362 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300365 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300367 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300370 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300372 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300375 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300378 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300380 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.302154 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300382 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300385 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300387 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300390 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300392 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300395 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300397 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.300399 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301203 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301211 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301214 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301217 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301219 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301222 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301225 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301227 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301230 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301233 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301236 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301238 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.302675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301241 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301243 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301246 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301249 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301251 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301254 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301257 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301260 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301262 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301265 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301268 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301270 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301273 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301275 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301278 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301280 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301283 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301286 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301288 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.303232 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301290 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301293 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301295 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301298 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301300 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301303 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301305 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301307 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301310 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301315 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301319 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301321 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301324 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301327 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301330 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301333 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301336 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301338 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301341 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.303697 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301343 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301346 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301349 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301351 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301353 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301356 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301359 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301361 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301364 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301366 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301369 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301371 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301374 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301376 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301378 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301381 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301383 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301386 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301388 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301391 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.304185 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301394 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301396 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301399 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301401 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301403 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301406 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301408 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301411 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301414 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301416 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301419 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301421 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301424 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301426 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301429 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301432 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301435 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301437 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301440 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.304709 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301442 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301445 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301448 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301450 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301453 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301455 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301457 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301460 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301463 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301465 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301470 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301473 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301476 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301479 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301481 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301483 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.301486 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302003 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302010 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.305341 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302013 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302017 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302020 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302023 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302026 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302028 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302031 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302034 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302037 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302039 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302042 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302045 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302047 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302050 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302053 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302055 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302058 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302060 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302063 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302065 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.305827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302067 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302085 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302088 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302091 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302095 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302098 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302101 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302104 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302107 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302109 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302112 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302115 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302117 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302120 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302122 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302125 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302128 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302130 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302133 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.306598 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302135 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302137 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302146 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302149 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302154 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302157 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302161 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302165 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302168 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302170 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302173 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302175 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302178 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302180 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302183 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302185 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302188 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302190 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302193 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302195 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.307056 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302198 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302200 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302203 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302206 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302208 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302210 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302213 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302215 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302218 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302220 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302223 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302225 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302227 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302230 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302232 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302240 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302243 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302246 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302248 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.307547 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302250 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302254 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302256 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302258 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302261 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302263 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302266 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302268 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302270 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302273 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302276 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302278 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302280 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302283 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302285 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302288 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302290 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302293 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302296 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.308008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302298 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302300 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302303 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302305 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302308 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302310 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.302313 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302424 4543 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302433 4543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302438 4543 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302450 4543 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302455 4543 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302459 4543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302463 4543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302467 4543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302470 4543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302473 4543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302476 4543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302479 4543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302482 4543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302485 4543 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302488 4543 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302491 4543 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:42:18.308494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302494 4543 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302497 4543 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302504 4543 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302507 4543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302510 4543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302513 4543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302517 4543 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302521 4543 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302523 4543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302526 4543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302530 4543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302533 4543 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302536 4543 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302539 4543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302542 4543 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302545 4543 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302549 4543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302552 4543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302555 4543 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302557 4543 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302568 4543 flags.go:64] FLAG: --enable-server="true" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302573 4543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302580 4543 flags.go:64] FLAG: --event-burst="100" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302583 4543 flags.go:64] FLAG: --event-qps="50" Oct 29 21:42:18.309242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302586 4543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302589 4543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302592 4543 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302596 4543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302599 4543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302602 4543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302605 4543 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302608 4543 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302611 4543 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302614 4543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302616 4543 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302619 4543 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302622 4543 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302625 4543 flags.go:64] FLAG: --feature-gates="" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302629 4543 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302632 4543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302635 4543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302638 4543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302641 4543 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302644 4543 flags.go:64] FLAG: --help="false" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302646 4543 flags.go:64] FLAG: --hostname-override="ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302649 4543 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302652 4543 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:42:18.309867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302655 4543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302658 4543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302662 4543 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302664 4543 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302667 4543 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302670 4543 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302679 4543 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302683 4543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302686 4543 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302689 4543 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302692 4543 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302695 4543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302698 4543 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302700 4543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302703 4543 flags.go:64] FLAG: --lock-file="" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302707 4543 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302710 4543 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302713 4543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302719 4543 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302721 4543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302724 4543 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302727 4543 flags.go:64] FLAG: --logging-format="text" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302730 4543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302733 4543 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:42:18.310440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302736 4543 flags.go:64] FLAG: --manifest-url="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302738 4543 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302743 4543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302746 4543 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302750 4543 flags.go:64] FLAG: --max-pods="110" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302752 4543 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302755 4543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302758 4543 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302761 4543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302764 4543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302767 4543 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302769 4543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302776 4543 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302779 4543 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302783 4543 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302792 4543 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302795 4543 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302798 4543 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302800 4543 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302803 4543 flags.go:64] FLAG: --port="10250" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302806 4543 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302809 4543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0728368544a028c6d" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302812 4543 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302816 4543 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302819 4543 flags.go:64] FLAG: --register-node="true" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302822 4543 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:42:18.311029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302825 4543 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302828 4543 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302831 4543 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302834 4543 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302838 4543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302840 4543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302843 4543 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302846 4543 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302849 4543 flags.go:64] FLAG: --runonce="false" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302852 4543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302855 4543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302858 4543 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302860 4543 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302863 4543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302866 4543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302869 4543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302872 4543 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302875 4543 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302878 4543 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302881 4543 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302884 4543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302887 4543 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302896 4543 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302899 4543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302904 4543 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:42:18.312204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302907 4543 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302909 4543 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302916 4543 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302919 4543 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302923 4543 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302926 4543 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302928 4543 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302931 4543 flags.go:64] FLAG: --v="2" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302935 4543 flags.go:64] FLAG: --version="false" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302939 4543 flags.go:64] FLAG: --vmodule="" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302943 4543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.302946 4543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303133 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303138 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303141 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303144 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303146 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303149 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303152 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303154 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303156 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303159 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.313281 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303161 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303164 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303166 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303169 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303172 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303175 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303178 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303181 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303189 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303192 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303194 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303197 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303199 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303202 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303205 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303208 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303210 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303213 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303215 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303218 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.313830 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303220 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303222 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303232 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303235 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303238 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303241 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303244 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303246 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303249 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303251 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303254 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303256 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303259 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303261 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303264 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303267 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303269 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303273 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303276 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303279 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.314360 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303281 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303291 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303294 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303297 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303299 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303301 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303305 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303308 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303310 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303313 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303315 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303318 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303320 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303322 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303325 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303328 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303330 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303332 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303335 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.315086 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303338 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303340 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303343 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303345 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303348 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303350 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303353 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303357 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303360 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303363 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303366 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303369 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303371 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303375 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303378 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303386 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303389 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303392 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303395 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.315657 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303398 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303401 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303403 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303406 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303408 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303410 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303413 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303415 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303418 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303420 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303423 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303425 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303427 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303430 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303432 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303435 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303437 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.316353 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303439 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.303467 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303514 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303518 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303521 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303524 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303526 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303530 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303533 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303535 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303538 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303540 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303548 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303551 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303554 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303558 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303561 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.317159 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303563 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303566 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303568 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303571 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303573 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303576 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303578 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303581 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303583 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303586 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303588 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303591 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303593 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303596 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303598 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303601 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303603 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303605 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303608 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.317916 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303610 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303613 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303616 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303619 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303622 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303625 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303628 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303630 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303634 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303644 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303647 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303651 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303654 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303656 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303660 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303663 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303666 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303669 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303671 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.318629 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303674 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303676 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303679 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303681 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303684 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303686 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303689 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303691 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303693 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303696 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303698 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303701 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303704 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303706 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303709 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303712 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303715 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303718 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303720 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303723 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.319254 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303726 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303728 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303730 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303738 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303742 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303745 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303747 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303750 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303752 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303755 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303757 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303760 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303762 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303765 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303767 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303770 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303772 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303775 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303777 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.319782 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303780 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303782 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303785 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303787 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303789 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303792 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303794 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303797 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303799 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303804 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303807 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303809 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303813 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.303815 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.308195 4543 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.308210 4543 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.308232 4543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.308237 4543 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308307 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308312 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.320258 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308315 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308318 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308321 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308324 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308326 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308329 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308331 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308334 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308336 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308339 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308342 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308344 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308347 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308350 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308352 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308354 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308357 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308360 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308363 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308365 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.320747 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308368 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308370 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308373 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308376 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308378 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308381 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308383 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308386 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308388 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308398 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308401 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308403 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308405 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308408 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308410 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308413 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308415 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308417 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308420 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.321323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308422 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308424 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308430 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308433 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308436 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308438 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308441 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308443 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308446 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308449 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308451 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308454 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308456 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308459 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308462 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308481 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308485 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308489 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308492 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308495 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.321789 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308498 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308501 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308504 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308512 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308515 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308517 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308520 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308522 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308525 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308527 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308530 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308532 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308534 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308537 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308539 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308542 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308544 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308547 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308549 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308552 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.322298 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308555 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308557 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308560 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308563 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308567 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308570 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308572 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308575 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308577 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308579 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308582 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308584 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308587 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308589 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308592 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308594 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308602 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308605 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308607 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.322801 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308609 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308612 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308616 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308619 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308622 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308625 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.308653 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308828 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308832 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308835 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308852 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308856 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308859 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308871 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308875 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.323388 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308878 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308880 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308883 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308886 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308888 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308891 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308893 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308896 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308898 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308901 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308903 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308906 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308908 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308910 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308913 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308921 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308924 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308927 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308929 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308932 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.323818 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308934 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308936 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308939 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308941 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308944 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308947 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308949 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308952 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308954 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308957 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308959 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308961 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308964 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308966 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308968 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308971 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308973 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308975 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308978 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308980 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.324488 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308983 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308985 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308988 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308990 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308992 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308995 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.308997 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309000 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309007 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309010 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309012 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309015 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309017 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309021 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309025 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309028 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309031 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309033 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309036 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.325228 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309038 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309041 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309043 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309046 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309048 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309050 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309053 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309055 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309058 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309060 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309062 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309065 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309067 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309070 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309094 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309097 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309100 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309104 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309107 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309111 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.325877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309114 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309117 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309126 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309128 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309131 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309133 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309136 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309138 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309140 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309143 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309145 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309148 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309150 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309152 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309155 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309158 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309160 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309163 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.326449 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309165 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.309190 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309236 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309240 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309243 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309246 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309248 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309251 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309253 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309256 4543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309258 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309261 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309269 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309278 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309281 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309284 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:18.326875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309286 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309294 4543 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309297 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309301 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309304 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309307 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309310 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309313 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309316 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309318 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309321 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309323 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309325 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309328 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309330 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309333 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309335 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309338 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309340 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:18.327319 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309342 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309345 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309347 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309350 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309352 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309355 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309357 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309359 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309362 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309364 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309369 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309371 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309374 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309376 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309378 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309386 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309389 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309391 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309393 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:18.327847 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309396 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309398 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309400 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309404 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309407 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309409 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309412 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309414 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309417 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309419 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309421 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309424 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309427 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309429 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309431 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309434 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309436 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309439 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309441 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:18.328596 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309444 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309446 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309449 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309452 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309456 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309458 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309460 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309463 4543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309465 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309468 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309475 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309478 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309481 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309483 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309486 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309488 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309491 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309494 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309496 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309499 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:18.329392 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309501 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309504 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309507 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309509 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309512 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309514 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309517 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309519 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309521 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309525 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309528 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309531 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309534 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309537 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:18.309540 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.309586 4543 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.310535 4543 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.310596 4543 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.310789 4543 server.go:1014] "Starting client certificate rotation" Oct 29 21:42:18.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.310945 4543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.311828 4543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:37 +0000 UTC" deadline="2028-06-14 16:23:50.488479882 +0000 UTC" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.311884 4543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14250h41m32.176600727s" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.312434 4543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.314394 4543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.315946 4543 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.318031 4543 log.go:25] "Validated CRI v1 image API" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.318815 4543 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:42:18.331000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.320283 4543 fs.go:135] Filesystem UUIDs: map[220c75d5-1746-4dec-8987-ed758625d6a6:/dev/nvme0n1p4 5ed825ee-1838-494a-837b-a5195f63849e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 29 21:42:18.331389 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.320303 4543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fa5427f65461829783d2c970a08af912655ce5bfed29b540466f4b5a39ee3e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fa5427f65461829783d2c970a08af912655ce5bfed29b540466f4b5a39ee3e8/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b49a2a6-8054-4068-bd32-fdd07397691a/volumes/kubernetes.io~projected/kube-api-access-557nl:{mountpoint:/var/lib/kubelet/pods/0b49a2a6-8054-4068-bd32-fdd07397691a/volumes/kubernetes.io~projected/kube-api-access-557nl major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16564c0f-cddb-477c-95e4-e9d3a9996bd2/volumes/kubernetes.io~projected/kube-api-access-vtffq:{mountpoint:/var/lib/kubelet/pods/16564c0f-cddb-477c-95e4-e9d3a9996bd2/volumes/kubernetes.io~projected/kube-api-access-vtffq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba/volumes/kubernetes.io~projected/kube-api-access-pnw6x:{mountpoint:/var/lib/kubelet/pods/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba/volumes/kubernetes.io~projected/kube-api-access-pnw6x major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29210f25-02b2-48e3-9d5b-24fbf755276a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/29210f25-02b2-48e3-9d5b-24fbf755276a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332737e0-8635-454a-942a-2ba3ca23e966/volumes/kubernetes.io~projected/kube-api-access-5drz2:{mountpoint:/var/lib/kubelet/pods/332737e0-8635-454a-942a-2ba3ca23e966/volumes/kubernetes.io~projected/kube-api-access-5drz2 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~projected/kube-api-access-47szc:{mountpoint:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~projected/kube-api-access-47szc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~projected/kube-api-access-pkm4f:{mountpoint:/var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~projected/kube-api-access-pkm4f major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/427f41b3-dae2-47cc-a0f1-749c57377d7e/volumes/kubernetes.io~projected/kube-api-access-4dtv9:{mountpoint:/var/lib/kubelet/pods/427f41b3-dae2-47cc-a0f1-749c57377d7e/volumes/kubernetes.io~projected/kube-api-access-4dtv9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/507ad5fa-55a9-4719-aa6a-12a777a05c5e/volumes/kubernetes.io~projected/kube-api-access-kc4rr:{mountpoint:/var/lib/kubelet/pods/507ad5fa-55a9-4719-aa6a-12a777a05c5e/volumes/kubernetes.io~projected/kube-api-access-kc4rr major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5921e77d-c1b2-448a-bd09-fc72e935dfc7/volumes/kubernetes.io~projected/kube-api-access-mm5m4:{mountpoint:/var/lib/kubelet/pods/5921e77d-c1b2-448a-bd09-fc72e935dfc7/volumes/kubernetes.io~projected/kube-api-access-mm5m4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7168e02c-2f77-4dbe-bc05-64923e69869f/volumes/kubernetes.io~projected/kube-api-access-dsgc7:{mountpoint:/var/lib/kubelet/pods/7168e02c-2f77-4dbe-bc05-64923e69869f/volumes/kubernetes.io~projected/kube-api-access-dsgc7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7672c73e-69be-484f-84ae-12c26a104e59/volumes/kubernetes.io~projected/kube-api-access-nd8w4:{mountpoint:/var/lib/kubelet/pods/7672c73e-69be-484f-84ae-12c26a104e59/volumes/kubernetes.io~projected/kube-api-access-nd8w4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d32947-bedc-47aa-bd2e-48a33b4a309f/volumes/kubernetes.io~projected/kube-api-access-l6rln:{mountpoint:/var/lib/kubelet/pods/97d32947-bedc-47aa-bd2e-48a33b4a309f/volumes/kubernetes.io~projected/kube-api-access-l6rln major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~projected/kube-api-access-k46k8:{mountpoint:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~projected/kube-api-access-k46k8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3042dc3-787e-46c3-80b8-dfb9294328d7/volumes/kubernetes.io~projected/kube-api-access-5zp86:{mountpoint:/var/lib/kubelet/pods/a3042dc3-787e-46c3-80b8-dfb9294328d7/volumes/kubernetes.io~projected/kube-api-access-5zp86 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~projected/kube-api-access-pb8wq:{mountpoint:/var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~projected/kube-api-access-pb8wq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~secret/signing-key major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad6f194-bef0-48b9-bf0a-725194f1703a/volumes/kubernetes.io~projected/kube-api-access-pgncj:{mountpoint:/var/lib/kubelet/pods/fad6f194-bef0-48b9-bf0a-725194f1703a/volumes/kubernetes.io~projected/kube-api-access-pgncj major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c1d39c8e5414e9f56e356fd7abf470666f16327e824881f6c37d1b479c66fc7d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5540b37c486ed742de9a74e8b28829dc356f157d34b2220c3d9eb46a49ae8f8d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/28d687436d8b01d820e9dbed35e62346a032179d57e12468bb615a7842477c86/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/6ee151dc25018279f88cc7ded8a0a0414b73548ec86c896a29cffb8db26a883f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6d0c829f3f4f4039cc2cad9ec7ba624da32e1f4f0d89a48fb17269c347d76f83/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2fe0b5efa063c2e21835946d68d784f57bfa0130829a1f656a981cef5e46323c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/32d60b832038a3bf788583e4d9d6397ea267a5bca49d4b26aab9b87d914ed3dc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/413d6fe65cedf064df9aa66126098b6f381c82218eb1dbff239c268e845c8c08/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3a513995a394acd4e11572fb7f2d7abc3f10eacac046620e6c2549e25d9f356c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/384f4a24cee021deb40d841ebf2ef613b302227e8fc6d0c329c433acb028f962/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/aa7d6a65be48e9c0f2c42efd73e5d66b7f27554e8d93e1a92b45c011dab8ea30/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/5937cf7341337b50a621c762806fb1a7551cad89b567fb953ce60769c8cc51bd/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3e3230498163e72a39e490de87c85ee013a0b50d2615af2c87f51c80f515e3a1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/edf7b9b7293fca6091c365e3d634f4494091b2bf62cebd388e05d363666ccc14/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ed06c89f89280ff34447d2e9dd37c9a6b0a29f38e40e936d6feda24068a29f9e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d77c9385e044965b07792f043d8d0b2072ad3fe68b98489fa0715b50d8baa101/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9400379794c3baa7fadfbb4524533ee513e35e7866b164e235c3b2b49739a01a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e22794ecd33588f25ab77bcae69b9a8786a4240aef8759f90c72aa7f58802e21/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7b1665b6d066d0a34f9e87e946f4f5c10086a5ba37949491a796f5fa79a4caf3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/175b640fa9fdae89689b76abe15f7ba40400ddb90fe32a0c531d21b6dddcfc16/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fb26ae8ee871e81bebc056aa4b8d0b24b544648d587ad02bd334fdf504198641/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/4741a70263467529d877c47f364eee68ce41de27c65b50a8f632a50eb08c8674/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/73e2d53f44256f42947a153277bc6977a8650e5af9cb1a48005a47a3ed0fbd3b/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/ff270d49a142f87d48efa700a4fe80197c258d3dae36796225f26b35dd01cf4f/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/ab188ef5b4124e8ed64908b61dcca807f0a3a55fb635d937d7591769864044ed/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/2421807da6ca370ae78de75d067f3ca7b43016f5596dc3f0186b54c1b28d02b7/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/921f6aa9c2e92be1270112d230923749ae33906812980319e753cade2875199b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/a3df4f53d8d48a7fb1cf09c12b414d4cb098638518fafc335c707dd9be948b1b/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/fc209376664383ca7ba7e647527d55c3c23b55a0178facc5e55701594fee621c/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d24c70a9e1074d94052f5368ee7b1530e5ef8bf78fd484076c5645f959ec2e8f/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e71abdef8b2a14b855c60f24db3b87b07238c186f909f95150e5e9201e595395/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/ef96642b1ead9d477f7f2a1f5232ae7ce05873d28e47d8e0b2c5efc060cb148c/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/c236e4d7644a820fef294a4ba7f708a15a05bddad52677df0f38ab4bdf47a6dd/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/b87aaf3bb0d9ef5eb684d5465c7bb338eb45c005d9deac64002c88d1b3c54346/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e56a0693b09f22d83e8d594bc729fa9328a31edc5d74da2d70d43e27d0f04fb6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f925caed2c4d0797269ebfd840fc34b911f05b7145ed2c6abb265be4aa3a6238/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dc09c7729ca66ff71f62d9523feb83132a9dd5336184394540e0ec8d3e3ac3c8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bb04071a9f866b37ec6004295d7c557bede2a68d0a5591ec0863833170c08b4d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/311ba04041ebfdbe30f14ccfec321ff4d7cdca4f11df70b194ca970bbc5c1737/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fb60cf6a5401a40e97d9fe117251370cc535b5f76411b8893cf7679a84bcd4a9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ae72522566160a2d6bbfa527f5f3737b014aecd37bae6ea7e62de93543986208/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e196e05063eb1d4d04d0843e63a220a3f2635082b3be62e2a1a535c7268d5649/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/911940c7dbda56b59d714c6ffd099c50a6e2567bb3c928de849632cc9c78faf2/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.326412 4543 manager.go:217] Machine: {Timestamp:2026-10-29 21:42:18.325926356 +0000 UTC m=+0.067857020 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098839 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a2da4e5ab56d77ec34e42d6cc122 SystemUUID:ec26a2da-4e5a-b56d-77ec-34e42d6cc122 BootID:ca15e967-4c3e-4088-af4a-29adfeb85284 Filesystems:[{Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad6f194-bef0-48b9-bf0a-725194f1703a/volumes/kubernetes.io~projected/kube-api-access-pgncj DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/618d94647bab39116d1de8e2cdcbfe39b41401505da22bc126715a8c0b89741f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1405e5f9867f20e2d27d05d797caa934ced6f5d796b212b18bf3a3c7df6562bd/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/507ad5fa-55a9-4719-aa6a-12a777a05c5e/volumes/kubernetes.io~projected/kube-api-access-kc4rr DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/427f41b3-dae2-47cc-a0f1-749c57377d7e/volumes/kubernetes.io~projected/kube-api-access-4dtv9 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/332737e0-8635-454a-942a-2ba3ca23e966/volumes/kubernetes.io~projected/kube-api-access-5drz2 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b49a2a6-8054-4068-bd32-fdd07397691a/volumes/kubernetes.io~projected/kube-api-access-557nl DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59ce82f2f73f185861bd4397fd5663576b908330a9c11bb7c0f1eb9dcd525155/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volumes/kubernetes.io~projected/kube-api-access-47szc DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5921e77d-c1b2-448a-bd09-fc72e935dfc7/volumes/kubernetes.io~projected/kube-api-access-mm5m4 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fa5427f65461829783d2c970a08af912655ce5bfed29b540466f4b5a39ee3e8/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40b2f40e5398cec356dba3417c28cc93ae03ce3360dd3283d7fbaca0cb5851fe/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97d32947-bedc-47aa-bd2e-48a33b4a309f/volumes/kubernetes.io~projected/kube-api-access-l6rln DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16564c0f-cddb-477c-95e4-e9d3a9996bd2/volumes/kubernetes.io~projected/kube-api-access-vtffq DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba/volumes/kubernetes.io~projected/kube-api-access-pnw6x DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36a1b6de95a2d1613500798a5d5dddaf6f4fc8faf3c92bb3b2e46fd2913dbfce/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~projected/kube-api-access-pkm4f DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7672c73e-69be-484f-84ae-12c26a104e59/volumes/kubernetes.io~projected/kube-api-access-nd8w4 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~projected/kube-api-access-k46k8 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c6522dbcc9e3f68141f9353825e6d60339a4225fc2182417ef6b739d6afb80d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/340734d7-e5e9-440d-8d36-d88ab0c4bf83/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4ad9b3d71ddeb0e340c712a93e77bffa5b59ad9805d048334496e8f50f45bae/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5319358d73168f005314e560bf0300fbd9a54a491ac8969cd086f576d986fd0/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29210f25-02b2-48e3-9d5b-24fbf755276a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41822b6b59977797a31ddb71343720f30c7e9f1ef6900fa9fe8400ef8ad32cb1/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d293727f087dc521a4f53e2f8aff9d5c3a29b2a2e287280eb1d717833337a672/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~projected/kube-api-access-pb8wq DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b9c1f3d5f4b6b88e10dcf26dcd342c3f3629707f465e3c619802d9c2ddeccd9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f7aa62f-dc8e-4339-bfa0-7553135f02c7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3042dc3-787e-46c3-80b8-dfb9294328d7/volumes/kubernetes.io~projected/kube-api-access-5zp86 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ac12563ce34071ecdc259d4d615bf8987248c44dcadaa6c7425af6e91060e04/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7168e02c-2f77-4dbe-bc05-64923e69869f/volumes/kubernetes.io~projected/kube-api-access-dsgc7 DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e2acd49a-da89-47ed-b1cb-40e068e16d85/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38d4cae5-2c19-4166-9d60-4fcc17b689b6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:40b2f40e5398cec MacAddress:b2:1d:23:a9:fa:d6 Speed:10000 Mtu:8901} {Name:41822b6b5997779 MacAddress:0a:07:25:72:28:ff Speed:10000 Mtu:8901} {Name:6c6522dbcc9e3f6 MacAddress:06:69:ae:f3:dd:d3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f5:b3:6b:15 Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:12:d1:aa:52:74 Speed:0 Mtu:8901} {Name:d293727f087dc52 MacAddress:86:b6:91:ce:d1:6b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f5:b3:6b:15 Speed:0 Mtu:9001} {Name:f008998401fde0e MacAddress:22:09:d5:ee:00:1d Speed:10000 Mtu:8901} {Name:f5319358d73168f MacAddress:16:b7:99:f8:99:5f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:49:c0:80:21:b7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:86:b0:04:59:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.326704 4543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.326741 4543 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327005 4543 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327028 4543 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327165 4543 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327173 4543 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327183 4543 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327195 4543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327276 4543 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327412 4543 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327511 4543 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327542 4543 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327550 4543 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327558 4543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327566 4543 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.327573 4543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.330172 4543 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.330372 4543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:18.331635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.330827 4543 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:42:18.333364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333347 4543 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:18.333364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333362 4543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:18.333492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333468 4543 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 21:42:18.333681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333666 4543 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:42:18.333740 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333706 4543 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:42:18.333828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333816 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333836 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333846 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333855 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333864 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333873 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:42:18.333887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333883 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333893 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333904 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333913 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333926 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333941 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333972 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:42:18.334188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.333981 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:42:18.337238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.337222 4543 server.go:1263] "Started kubelet" Oct 29 21:42:18.337321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.337293 4543 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:42:18.340098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.337428 4543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:42:18.340098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.337490 4543 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:42:18.340098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.337886 4543 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:42:18.340098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.338438 4543 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:42:18.337753 ip-10-0-128-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:42:18.340412 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.340255 4543 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:42:18.340530 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.340487 4543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:42:18.341539 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.340764 4543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-100.ec2.internal\" not found" Oct 29 21:42:18.341539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.340820 4543 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:42:18.341539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.340827 4543 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:42:18.341539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.341187 4543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:42:18.342724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.341626 4543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:39 +0000 UTC" deadline="2028-06-29 19:13:34.164312946 +0000 UTC" Oct 29 21:42:18.342724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.341659 4543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14613h31m15.822656961s" Oct 29 21:42:18.342724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.342193 4543 factory.go:55] Registering systemd factory Oct 29 21:42:18.342724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.342208 4543 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.342882 4543 factory.go:153] Registering CRI-O factory Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.342905 4543 factory.go:223] Registration of the crio container factory successfully Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.343032 4543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.343059 4543 factory.go:103] Registering Raw factory Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.343124 4543 manager.go:1199] Started watching for new ooms in manager Oct 29 21:42:18.344317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.343578 4543 manager.go:319] Starting recovery of all containers Oct 29 21:42:18.350397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350339 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f7aa62f-dc8e-4339-bfa0-7553135f02c7" volumeName="kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350402 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2acd49a-da89-47ed-b1cb-40e068e16d85" volumeName="kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350417 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad6f194-bef0-48b9-bf0a-725194f1703a" volumeName="kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350436 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b49a2a6-8054-4068-bd32-fdd07397691a" volumeName="kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350450 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b49a2a6-8054-4068-bd32-fdd07397691a" volumeName="kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350469 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b49a2a6-8054-4068-bd32-fdd07397691a" volumeName="kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350482 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340734d7-e5e9-440d-8d36-d88ab0c4bf83" volumeName="kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc" seLinuxMountContext="" Oct 29 21:42:18.350504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350496 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332737e0-8635-454a-942a-2ba3ca23e966" volumeName="kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350518 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5921e77d-c1b2-448a-bd09-fc72e935dfc7" volumeName="kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350531 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5921e77d-c1b2-448a-bd09-fc72e935dfc7" volumeName="kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350549 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5921e77d-c1b2-448a-bd09-fc72e935dfc7" volumeName="kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350564 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7168e02c-2f77-4dbe-bc05-64923e69869f" volumeName="kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350581 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16564c0f-cddb-477c-95e4-e9d3a9996bd2" volumeName="kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350593 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="427f41b3-dae2-47cc-a0f1-749c57377d7e" volumeName="kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350615 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2acd49a-da89-47ed-b1cb-40e068e16d85" volumeName="kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350629 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2889a3a6-52aa-45d9-a13c-d4fb5cf6abba" volumeName="kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350642 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29210f25-02b2-48e3-9d5b-24fbf755276a" volumeName="kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350661 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340734d7-e5e9-440d-8d36-d88ab0c4bf83" volumeName="kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350674 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340734d7-e5e9-440d-8d36-d88ab0c4bf83" volumeName="kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350692 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="427f41b3-dae2-47cc-a0f1-749c57377d7e" volumeName="kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350705 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="427f41b3-dae2-47cc-a0f1-749c57377d7e" volumeName="kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350718 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7672c73e-69be-484f-84ae-12c26a104e59" volumeName="kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350734 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f7aa62f-dc8e-4339-bfa0-7553135f02c7" volumeName="kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350748 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38d4cae5-2c19-4166-9d60-4fcc17b689b6" volumeName="kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350767 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="507ad5fa-55a9-4719-aa6a-12a777a05c5e" volumeName="kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350779 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7168e02c-2f77-4dbe-bc05-64923e69869f" volumeName="kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350794 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3042dc3-787e-46c3-80b8-dfb9294328d7" volumeName="kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350812 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad6f194-bef0-48b9-bf0a-725194f1703a" volumeName="kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350838 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340734d7-e5e9-440d-8d36-d88ab0c4bf83" volumeName="kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350852 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="427f41b3-dae2-47cc-a0f1-749c57377d7e" volumeName="kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350869 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7672c73e-69be-484f-84ae-12c26a104e59" volumeName="kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350882 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d32947-bedc-47aa-bd2e-48a33b4a309f" volumeName="kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350894 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4f0a27b-37b7-4dc0-932d-913d95a9d0e4" volumeName="kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350911 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2acd49a-da89-47ed-b1cb-40e068e16d85" volumeName="kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350924 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b49a2a6-8054-4068-bd32-fdd07397691a" volumeName="kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350946 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332737e0-8635-454a-942a-2ba3ca23e966" volumeName="kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350959 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f7aa62f-dc8e-4339-bfa0-7553135f02c7" volumeName="kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8" seLinuxMountContext="" Oct 29 21:42:18.350969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350972 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3042dc3-787e-46c3-80b8-dfb9294328d7" volumeName="kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script" seLinuxMountContext="" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.350990 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4f0a27b-37b7-4dc0-932d-913d95a9d0e4" volumeName="kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca" seLinuxMountContext="" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.351003 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332737e0-8635-454a-942a-2ba3ca23e966" volumeName="kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2" seLinuxMountContext="" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.351019 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="340734d7-e5e9-440d-8d36-d88ab0c4bf83" volumeName="kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides" seLinuxMountContext="" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.351035 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38d4cae5-2c19-4166-9d60-4fcc17b689b6" volumeName="kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert" seLinuxMountContext="" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.351048 4543 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:42:18.351964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.351055 4543 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:42:18.375736 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.375700 4543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:42:18.377109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.377063 4543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:42:18.377109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.377111 4543 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:42:18.377270 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.377133 4543 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:42:18.377270 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.377177 4543 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 21:42:18.394037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.394016 4543 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 21:42:18.395036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395018 4543 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:18.395341 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395321 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="e187ae09f1d2f930111a664e1459898a6692d6526b6452ef10811eb65ef27035" exitCode=0 Oct 29 21:42:18.395395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395343 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="0180a39e5637c6737c2b82fdf69889686f6d768f0c6a8467d1ad661c4462c3f5" exitCode=0 Oct 29 21:42:18.395395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395351 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="23a0f69ea8e3ae84b49ed1f47b384c38fc14b3f919d3e636d7cbeeeedfed137e" exitCode=0 Oct 29 21:42:18.395395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395359 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="b4fc24628bfe0172ada8e1c55498e7afd1079668d22359747e83b6e17abc51ab" exitCode=0 Oct 29 21:42:18.395395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395364 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="667919cfae05369d7db5cdd56b168c2b9c4960eda857a0cf0ed49852d5530ed6" exitCode=0 Oct 29 21:42:18.395395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.395369 4543 generic.go:356] "Generic (PLEG): container finished" podID="427f41b3-dae2-47cc-a0f1-749c57377d7e" containerID="2d3c1e51497063011619f9ded17a3958ebaf3599ad45b63ebb3e740907d66be1" exitCode=0 Oct 29 21:42:18.396698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.396507 4543 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:18.409146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.409128 4543 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:18.409230 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.409179 4543 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:18.410111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.410059 4543 generic.go:356] "Generic (PLEG): container finished" podID="186cd4ee51f579a3597049aa602c4c28" containerID="9153a8c88cb7d47d0af11b3ddb6cd5488470a7b4de7e0453f6b917f3566680eb" exitCode=0 Oct 29 21:42:18.424639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.424596 4543 manager.go:324] Recovery completed Oct 29 21:42:18.439175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439158 4543 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:42:18.439175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439171 4543 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:42:18.439303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439186 4543 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439306 4543 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439317 4543 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439332 4543 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439337 4543 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439342 4543 policy_none.go:50] "Start" Oct 29 21:42:18.439350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439350 4543 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:42:18.439522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439359 4543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:42:18.439522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439476 4543 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 21:42:18.439522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439483 4543 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 21:42:18.439522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.439488 4543 policy_none.go:44] "Start" Oct 29 21:42:18.443334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.443286 4543 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:42:18.445897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.445883 4543 manager.go:341] "Starting Device Plugin manager" Oct 29 21:42:18.446397 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.445907 4543 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:42:18.446448 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446401 4543 server.go:87] "Starting device plugin registration server" Oct 29 21:42:18.446639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446629 4543 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:42:18.446681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446640 4543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:42:18.447936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446736 4543 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:42:18.447936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446867 4543 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:42:18.447936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446898 4543 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:42:18.447936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446905 4543 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:42:18.447936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.446943 4543 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:42:18.446889183Z","UUID":"664d457a-9b90-48b7-b420-332a6542eddf","Handler":null,"Name":"","Endpoint":""} Oct 29 21:42:18.448335 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.448110 4543 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:42:18.448501 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.448473 4543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:42:18.448501 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.448501 4543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:42:18.477574 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.477542 4543 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal"] Oct 29 21:42:18.478017 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.477984 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" event={"ID":"2787fae5ca7f0b39f010401de9cba34b","Type":"ContainerStarted","Data":"64702d90fc08be7e1a6fdddfdeb073bbb7c3526bb3e1468df031a6b8a0ce8d07"} Oct 29 21:42:18.478059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478029 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" event={"ID":"2787fae5ca7f0b39f010401de9cba34b","Type":"ContainerStarted","Data":"eae5c2b9210c7e81012a4cfecd44dbe10d6097a262fc7639ec0568c4c6cea959"} Oct 29 21:42:18.478139 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478122 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerStarted","Data":"4405e53be90654dd9618dd76fb755e27802f66985487f32fdb9128c52e46d452"} Oct 29 21:42:18.478194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478142 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerDied","Data":"9153a8c88cb7d47d0af11b3ddb6cd5488470a7b4de7e0453f6b917f3566680eb"} Oct 29 21:42:18.478194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478159 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" event={"ID":"186cd4ee51f579a3597049aa602c4c28","Type":"ContainerStarted","Data":"a90aecd7bb7389696b43eea94cdabfd4e6c72b7735d5ee0172ceed607b2b9d8c"} Oct 29 21:42:18.478281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478224 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.478281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.478237 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.502946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.502928 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:18.503008 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.502965 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.505572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.505561 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:18.505618 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:18.505586 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.547552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.547534 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:42:18.548706 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.548688 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:42:18.548804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.548717 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:42:18.548804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.548731 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:42:18.548804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.548760 4543 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.591179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.591161 4543 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.591230 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.591216 4543 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.594753 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.594735 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.594822 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.594760 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.594822 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.594775 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695243 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695243 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695209 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695243 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695231 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695256 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695280 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186cd4ee51f579a3597049aa602c4c28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" (UID: \"186cd4ee51f579a3597049aa602c4c28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:18.695381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:18.695293 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2787fae5ca7f0b39f010401de9cba34b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" (UID: \"2787fae5ca7f0b39f010401de9cba34b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:19.328653 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.328605 4543 apiserver.go:52] "Watching apiserver" Oct 29 21:42:19.381778 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.381757 4543 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:42:19.382223 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.382204 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh","openshift-ovn-kubernetes/ovnkube-node-jj7md","openshift-cluster-node-tuning-operator/tuned-58pn4","kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal","openshift-image-registry/node-ca-wkxjk","openshift-multus/multus-wpkwp","openshift-service-ca/service-ca-5fffc97bdf-r9mdg","kube-system/konnectivity-agent-znvhf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95","openshift-multus/network-metrics-daemon-jvnws","openshift-insights/insights-runtime-extractor-jpxgr","openshift-ingress-canary/ingress-canary-hzqgn","openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh","openshift-multus/multus-additional-cni-plugins-htxp7","openshift-network-diagnostics/network-check-target-2thvl","openshift-network-operator/iptables-alerter-8j8nv","openshift-dns/node-resolver-2k6br","kube-system/global-pull-secret-syncer-2tdpb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal","openshift-dns/dns-default-p75bp"] Oct 29 21:42:19.387316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.387297 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:19.389099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.389060 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.389458 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.389443 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.399170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399142 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.399356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399339 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:42:19.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399356 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399358 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399377 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:42:19.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399373 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:42:19.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399427 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:42:19.399651 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.399431 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:42:19.408033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408016 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:42:19.408366 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408351 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.408441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408371 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.408652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408621 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:19.408652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408629 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:42:19.408652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408630 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408634 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408668 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408670 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408664 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408687 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:42:19.408832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.408825 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:42:19.409158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409144 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:42:19.409158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409148 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.409257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409243 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c7spd\"" Oct 29 21:42:19.409353 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409338 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:42:19.409396 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409389 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 21:42:19.409432 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409390 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.409432 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409391 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:19.409486 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409457 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:19.409654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409642 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sjp55\"" Oct 29 21:42:19.409848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.409835 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.411206 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.411194 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 21:42:19.411928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.411915 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:19.412554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.412540 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:42:19.413230 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413217 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:19.413282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413250 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413475 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:19.413491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413478 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413493 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.413739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413514 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413556 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413700 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413757 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.413976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413961 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:42:19.414041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413975 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.414041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.413999 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:19.414041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.414007 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:42:19.414221 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.414207 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:19.414269 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.414245 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.414269 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.414248 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.415033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415017 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415019 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415049 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415060 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415032 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415068 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:19.415119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415117 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.415299 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.415144 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 21:42:19.420400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.420384 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:42:19.420400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.420394 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:19.420510 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.420446 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:19.420734 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.420715 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:19.423181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.423158 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 21:42:19.424050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.424032 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:19.424459 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.424439 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:19.443164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.443150 4543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:42:19.446571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.446556 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:19.446631 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.446599 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-100.ec2.internal" Oct 29 21:42:19.447677 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.447652 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:19.447758 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.447692 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-100.ec2.internal" Oct 29 21:42:19.498248 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498224 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:19.498324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498251 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.498324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.498324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498283 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.498324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498298 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.498543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498526 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.498604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498574 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.498604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498599 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.498711 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498692 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.498804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498730 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.498804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498765 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4rr\" (UniqueName: \"kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr\") pod \"migrator-66df5d496b-f9wjh\" (UID: \"507ad5fa-55a9-4719-aa6a-12a777a05c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:19.498905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498810 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-daemon-config\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.498905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498837 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.498905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498834 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.498905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.498905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498903 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498919 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498934 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.498995 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499059 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5m4\" (UniqueName: \"kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zp86\" (UniqueName: \"kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.499135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499125 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fad6f194-bef0-48b9-bf0a-725194f1703a-serviceca\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499140 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499161 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-script-lib\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499171 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8w4\" (UniqueName: \"kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499199 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-key\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499214 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499281 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.499336 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499353 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499409 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499449 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499508 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47szc\" (UniqueName: \"kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499533 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkm4f\" (UniqueName: \"kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499533 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29210f25-02b2-48e3-9d5b-24fbf755276a-original-pull-secret\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499556 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499582 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.499661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499618 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499686 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499702 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxdl2\"" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499705 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovnkube-config\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499735 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499739 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499793 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499864 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499961 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499982 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.499996 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500039 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500094 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3042dc3-787e-46c3-80b8-dfb9294328d7-iptables-alerter-script\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500134 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500168 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46k8\" (UniqueName: \"kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500200 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500225 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-tuned\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500264 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500307 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332737e0-8635-454a-942a-2ba3ca23e966-tmp-dir\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8wq\" (UniqueName: \"kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500417 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500448 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500462 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500499 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500553 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7168e02c-2f77-4dbe-bc05-64923e69869f-data-volume\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500594 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500609 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:19.500627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500643 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/340734d7-e5e9-440d-8d36-d88ab0c4bf83-ovn-node-metrics-cert\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500677 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332737e0-8635-454a-942a-2ba3ca23e966-config-volume\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500740 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500751 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500766 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500800 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500832 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-agent-certs\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500841 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/340734d7-e5e9-440d-8d36-d88ab0c4bf83-env-overrides\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500893 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-tmp\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500898 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500921 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4f0a27b-37b7-4dc0-932d-913d95a9d0e4-konnectivity-ca\") pod \"konnectivity-agent-znvhf\" (UID: \"a4f0a27b-37b7-4dc0-932d-913d95a9d0e4\") " pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500927 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.500946 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501023 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtffq\" (UniqueName: \"kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501055 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501111 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501147 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501171 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501183 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.501294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drz2\" (UniqueName: \"kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501350 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501367 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501382 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501397 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501412 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgc7\" (UniqueName: \"kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501469 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgncj\" (UniqueName: \"kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501488 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501502 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501519 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501533 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501548 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501658 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnw6x\" (UniqueName: \"kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501708 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7672c73e-69be-484f-84ae-12c26a104e59-tmp-dir\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501711 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2acd49a-da89-47ed-b1cb-40e068e16d85-signing-cabundle\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501740 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d4cae5-2c19-4166-9d60-4fcc17b689b6-cert\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501749 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cni-binary-copy\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501745 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501798 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501825 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501903 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501939 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtv9\" (UniqueName: \"kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501969 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.501994 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502067 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.502062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502088 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.503019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502121 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.503019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502145 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.503019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502171 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.503019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.502334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/427f41b3-dae2-47cc-a0f1-749c57377d7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.521164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521140 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zp86\" (UniqueName: \"kubernetes.io/projected/a3042dc3-787e-46c3-80b8-dfb9294328d7-kube-api-access-5zp86\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.521274 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521143 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4rr\" (UniqueName: \"kubernetes.io/projected/507ad5fa-55a9-4719-aa6a-12a777a05c5e-kube-api-access-kc4rr\") pod \"migrator-66df5d496b-f9wjh\" (UID: \"507ad5fa-55a9-4719-aa6a-12a777a05c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f9wjh" Oct 29 21:42:19.521386 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521369 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rln\" (UniqueName: \"kubernetes.io/projected/97d32947-bedc-47aa-bd2e-48a33b4a309f-kube-api-access-l6rln\") pod \"network-check-target-2thvl\" (UID: \"97d32947-bedc-47aa-bd2e-48a33b4a309f\") " pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:19.521436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5m4\" (UniqueName: \"kubernetes.io/projected/5921e77d-c1b2-448a-bd09-fc72e935dfc7-kube-api-access-mm5m4\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.521660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521644 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47szc\" (UniqueName: \"kubernetes.io/projected/340734d7-e5e9-440d-8d36-d88ab0c4bf83-kube-api-access-47szc\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.521774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.521759 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8w4\" (UniqueName: \"kubernetes.io/projected/7672c73e-69be-484f-84ae-12c26a104e59-kube-api-access-nd8w4\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.522400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.522384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8wq\" (UniqueName: \"kubernetes.io/projected/e2acd49a-da89-47ed-b1cb-40e068e16d85-kube-api-access-pb8wq\") pod \"service-ca-5fffc97bdf-r9mdg\" (UID: \"e2acd49a-da89-47ed-b1cb-40e068e16d85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r9mdg" Oct 29 21:42:19.522607 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.522591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkm4f\" (UniqueName: \"kubernetes.io/projected/38d4cae5-2c19-4166-9d60-4fcc17b689b6-kube-api-access-pkm4f\") pod \"ingress-canary-hzqgn\" (UID: \"38d4cae5-2c19-4166-9d60-4fcc17b689b6\") " pod="openshift-ingress-canary/ingress-canary-hzqgn" Oct 29 21:42:19.523050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.523033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"periodic-gathering-8vksd-8q4gh\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:42:19.523312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.523295 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtffq\" (UniqueName: \"kubernetes.io/projected/16564c0f-cddb-477c-95e4-e9d3a9996bd2-kube-api-access-vtffq\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:19.523699 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.523682 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnw6x\" (UniqueName: \"kubernetes.io/projected/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kube-api-access-pnw6x\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.523893 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.523878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drz2\" (UniqueName: \"kubernetes.io/projected/332737e0-8635-454a-942a-2ba3ca23e966-kube-api-access-5drz2\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.524002 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.523985 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgncj\" (UniqueName: \"kubernetes.io/projected/fad6f194-bef0-48b9-bf0a-725194f1703a-kube-api-access-pgncj\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.524278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.524263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgc7\" (UniqueName: \"kubernetes.io/projected/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-api-access-dsgc7\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.525944 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.525929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtv9\" (UniqueName: \"kubernetes.io/projected/427f41b3-dae2-47cc-a0f1-749c57377d7e-kube-api-access-4dtv9\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.527840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.527821 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46k8\" (UniqueName: \"kubernetes.io/projected/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-kube-api-access-k46k8\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.590957 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.590907 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkxjk_fad6f194-bef0-48b9-bf0a-725194f1703a/node-ca/0.log" Oct 29 21:42:19.602682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602649 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.602758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602685 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.602758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602702 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.602758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3042dc3-787e-46c3-80b8-dfb9294328d7-host-slash\") pod \"iptables-alerter-8j8nv\" (UID: \"a3042dc3-787e-46c3-80b8-dfb9294328d7\") " pod="openshift-network-operator/iptables-alerter-8j8nv" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602752 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602794 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602800 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-kubelet\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602764 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-kubelet-config\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602866 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602896 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602913 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-conf\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.602916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602917 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602938 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602952 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-sys\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602956 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602955 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7672c73e-69be-484f-84ae-12c26a104e59-hosts-file\") pod \"node-resolver-2k6br\" (UID: \"7672c73e-69be-484f-84ae-12c26a104e59\") " pod="openshift-dns/node-resolver-2k6br" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602971 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.602978 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-var-lib-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603002 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-systemd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603011 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-node-log\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29210f25-02b2-48e3-9d5b-24fbf755276a-dbus\") pod \"global-pull-secret-syncer-2tdpb\" (UID: \"29210f25-02b2-48e3-9d5b-24fbf755276a\") " pod="kube-system/global-pull-secret-syncer-2tdpb" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603032 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603054 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-host\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603056 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603099 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603104 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-hostroot\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-netd\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603048 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-kubernetes\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603123 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-log-socket\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603126 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603148 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603152 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603162 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603183 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-socket-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603184 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603202 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603214 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603228 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.603233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603243 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603277 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603280 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-ovn\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603308 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-cni-bin\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603316 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-etc-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603320 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603325 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-systemd-units\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603356 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-k8s-cni-cncf-io\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603369 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-bin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603369 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-socket-dir-parent\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603365 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603392 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysconfig\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603414 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603416 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-registration-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603461 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7168e02c-2f77-4dbe-bc05-64923e69869f-crio-socket\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603482 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603497 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603511 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603525 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603539 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603554 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603570 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-run\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-var-lib-kubelet\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603599 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-systemd\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.603633 4543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603649 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-device-dir\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603630 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7168e02c-2f77-4dbe-bc05-64923e69869f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603605 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-system-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603673 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603600 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.603693 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:20.103675801 +0000 UTC m=+1.845606475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603723 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603747 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603728 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-sysctl-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603765 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603789 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603811 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-cni-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603812 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.603825 4543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603852 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-cnibin\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2889a3a6-52aa-45d9-a13c-d4fb5cf6abba-sys-fs\") pod \"aws-ebs-csi-driver-node-vbq95\" (UID: \"2889a3a6-52aa-45d9-a13c-d4fb5cf6abba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbq95" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603871 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603769 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-etc-kubernetes\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.603882 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:20.103862265 +0000 UTC m=+1.845792933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603896 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-kubelet\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603904 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-slash\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603935 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603951 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603971 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603977 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-run-openvswitch\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.604011 4543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604028 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.603998 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad6f194-bef0-48b9-bf0a-725194f1703a-host\") pod \"node-ca-wkxjk\" (UID: \"fad6f194-bef0-48b9-bf0a-725194f1703a\") " pod="openshift-image-registry/node-ca-wkxjk" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:19.604046 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs podName:16564c0f-cddb-477c-95e4-e9d3a9996bd2 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:20.104032965 +0000 UTC m=+1.845963634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs") pod "network-metrics-daemon-jvnws" (UID: "16564c0f-cddb-477c-95e4-e9d3a9996bd2") : secret "metrics-daemon-secret" not found Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604012 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-cnibin\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604063 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-multus-conf-dir\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604067 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604119 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604133 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604136 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-etc-modprobe-d\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604153 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-netns\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604155 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604177 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.604158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604192 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604208 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604230 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/340734d7-e5e9-440d-8d36-d88ab0c4bf83-host-run-ovn-kubernetes\") pod \"ovnkube-node-jj7md\" (UID: \"340734d7-e5e9-440d-8d36-d88ab0c4bf83\") " pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604232 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7aa62f-dc8e-4339-bfa0-7553135f02c7-lib-modules\") pod \"tuned-58pn4\" (UID: \"9f7aa62f-dc8e-4339-bfa0-7553135f02c7\") " pod="openshift-cluster-node-tuning-operator/tuned-58pn4" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604242 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-multus-certs\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-run-netns\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604269 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604269 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5921e77d-c1b2-448a-bd09-fc72e935dfc7-host-var-lib-cni-multus\") pod \"multus-wpkwp\" (UID: \"5921e77d-c1b2-448a-bd09-fc72e935dfc7\") " pod="openshift-multus/multus-wpkwp" Oct 29 21:42:19.606101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.604315 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/427f41b3-dae2-47cc-a0f1-749c57377d7e-system-cni-dir\") pod \"multus-additional-cni-plugins-htxp7\" (UID: \"427f41b3-dae2-47cc-a0f1-749c57377d7e\") " pod="openshift-multus/multus-additional-cni-plugins-htxp7" Oct 29 21:42:19.923097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:19.923025 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzqgn_38d4cae5-2c19-4166-9d60-4fcc17b689b6/serve-healthcheck-canary/0.log" Oct 29 21:42:20.107799 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.107773 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:20.107904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.107807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:20.107904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.107830 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:20.107972 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:20.107939 4543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:20.108026 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:20.108019 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:21.108005303 +0000 UTC m=+2.849936012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:20.108097 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:20.107939 4543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:20.108131 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:20.108124 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls podName:332737e0-8635-454a-942a-2ba3ca23e966 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:21.108110376 +0000 UTC m=+2.850041027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls") pod "dns-default-p75bp" (UID: "332737e0-8635-454a-942a-2ba3ca23e966") : secret "dns-default-metrics-tls" not found Oct 29 21:42:20.108203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.108184 4543 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:42:20.110554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.110535 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16564c0f-cddb-477c-95e4-e9d3a9996bd2-metrics-certs\") pod \"network-metrics-daemon-jvnws\" (UID: \"16564c0f-cddb-477c-95e4-e9d3a9996bd2\") " pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:20.288214 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.288158 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jvnws" Oct 29 21:42:20.589661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.589632 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jvnws"] Oct 29 21:42:20.592001 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:20.591967 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16564c0f_cddb_477c_95e4_e9d3a9996bd2.slice/crio-5cd6dfc13762b154e18f38e973aa0e248dfec4cc8216e5ee5c551cf119e2ec61 WatchSource:0}: Error finding container 5cd6dfc13762b154e18f38e973aa0e248dfec4cc8216e5ee5c551cf119e2ec61: Status 404 returned error can't find the container with id 5cd6dfc13762b154e18f38e973aa0e248dfec4cc8216e5ee5c551cf119e2ec61 Oct 29 21:42:20.594108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.594089 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:42:20.723231 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:20.723203 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-f9wjh_507ad5fa-55a9-4719-aa6a-12a777a05c5e/migrator/0.log" Oct 29 21:42:21.109879 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.109858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:21.115769 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.115734 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:21.115880 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.115780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:21.115880 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:21.115877 4543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:21.115989 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:21.115925 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls podName:7168e02c-2f77-4dbe-bc05-64923e69869f nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.115914521 +0000 UTC m=+4.857845172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpxgr" (UID: "7168e02c-2f77-4dbe-bc05-64923e69869f") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:21.118441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.118422 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332737e0-8635-454a-942a-2ba3ca23e966-metrics-tls\") pod \"dns-default-p75bp\" (UID: \"332737e0-8635-454a-942a-2ba3ca23e966\") " pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:21.189624 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.189597 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:21.430185 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.430113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jvnws" event={"ID":"16564c0f-cddb-477c-95e4-e9d3a9996bd2","Type":"ContainerStarted","Data":"5cd6dfc13762b154e18f38e973aa0e248dfec4cc8216e5ee5c551cf119e2ec61"} Oct 29 21:42:21.497983 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:21.497883 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p75bp"] Oct 29 21:42:21.506225 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:21.506187 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332737e0_8635_454a_942a_2ba3ca23e966.slice/crio-6eaa20048549101db598fff780bb91b7cd8b051e4485ebb5e05d3dab6952c0bf WatchSource:0}: Error finding container 6eaa20048549101db598fff780bb91b7cd8b051e4485ebb5e05d3dab6952c0bf: Status 404 returned error can't find the container with id 6eaa20048549101db598fff780bb91b7cd8b051e4485ebb5e05d3dab6952c0bf Oct 29 21:42:22.127316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:22.127258 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:22.127641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:22.127419 4543 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:42:22.435568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:22.435478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p75bp" event={"ID":"332737e0-8635-454a-942a-2ba3ca23e966","Type":"ContainerStarted","Data":"6eaa20048549101db598fff780bb91b7cd8b051e4485ebb5e05d3dab6952c0bf"} Oct 29 21:42:22.437534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:22.437498 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jvnws" event={"ID":"16564c0f-cddb-477c-95e4-e9d3a9996bd2","Type":"ContainerStarted","Data":"46f8c98333ab37444827f9214566d941b3e57d65f6579ae9bfdd85681ce40ece"} Oct 29 21:42:22.437665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:22.437550 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jvnws" event={"ID":"16564c0f-cddb-477c-95e4-e9d3a9996bd2","Type":"ContainerStarted","Data":"5a207a0cd9e29bfde16fbf9c60da1c2da69ff761959a7b6a4fb113b438ee2f89"} Oct 29 21:42:23.129761 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.129734 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:23.131913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.131897 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7168e02c-2f77-4dbe-bc05-64923e69869f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpxgr\" (UID: \"7168e02c-2f77-4dbe-bc05-64923e69869f\") " pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:23.290579 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.290517 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpxgr" Oct 29 21:42:23.442828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.442797 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p75bp" event={"ID":"332737e0-8635-454a-942a-2ba3ca23e966","Type":"ContainerStarted","Data":"b1fdc262ced53c2e4a2fae5fb68e4121544ae8431a460335066a21fd884fa81f"} Oct 29 21:42:23.442921 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.442850 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p75bp" event={"ID":"332737e0-8635-454a-942a-2ba3ca23e966","Type":"ContainerStarted","Data":"edb7c9170f692b4197c798f2b9acf90357c87082f89a4c4d80cf4fdbb6319700"} Oct 29 21:42:23.587571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:23.587550 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpxgr"] Oct 29 21:42:23.590400 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:23.590372 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7168e02c_2f77_4dbe_bc05_64923e69869f.slice/crio-612d4bc0ab415cbe229002fa1d1916ad2c75343779315f02d671c3d5e77f443f WatchSource:0}: Error finding container 612d4bc0ab415cbe229002fa1d1916ad2c75343779315f02d671c3d5e77f443f: Status 404 returned error can't find the container with id 612d4bc0ab415cbe229002fa1d1916ad2c75343779315f02d671c3d5e77f443f Oct 29 21:42:24.452171 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:24.452134 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpxgr" event={"ID":"7168e02c-2f77-4dbe-bc05-64923e69869f","Type":"ContainerStarted","Data":"84a3adf585bc5ff246862f16544eaddc0f605316d2b25e86b626cc24110c90bc"} Oct 29 21:42:24.452534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:24.452210 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpxgr" event={"ID":"7168e02c-2f77-4dbe-bc05-64923e69869f","Type":"ContainerStarted","Data":"5335744f852b409137c07e576938f214f0f23e13444b8c5026f4c8143dbfedcd"} Oct 29 21:42:24.452534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:24.452230 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpxgr" event={"ID":"7168e02c-2f77-4dbe-bc05-64923e69869f","Type":"ContainerStarted","Data":"612d4bc0ab415cbe229002fa1d1916ad2c75343779315f02d671c3d5e77f443f"} Oct 29 21:42:24.452534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:24.452252 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:26.459777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:26.459741 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpxgr" event={"ID":"7168e02c-2f77-4dbe-bc05-64923e69869f","Type":"ContainerStarted","Data":"da81b640f3c8ff85facaf6b21a345343763f33f37fc18c66d48bf7936b87cd1f"} Oct 29 21:42:26.998420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:26.998387 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:27.958920 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:27.958888 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:42:28.157789 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:28.157765 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jj7md" Oct 29 21:42:33.309660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.309626 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56c44567bf-hdhbj"] Oct 29 21:42:33.311728 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.311712 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.367563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.367531 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 21:42:33.367685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.367620 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g25n\"" Oct 29 21:42:33.367685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.367635 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 21:42:33.367794 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.367730 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 21:42:33.367848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.367793 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 21:42:33.368485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.368464 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c44567bf-hdhbj"] Oct 29 21:42:33.393238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393215 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-image-registry-private-configuration\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393343 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393244 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-certificates\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393343 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-trusted-ca\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393343 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393282 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-bound-sa-token\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393458 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393352 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-installation-pull-secrets\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393458 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393414 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd24f7c5-67b5-49fe-8691-06b90349e21d-ca-trust-extracted\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393458 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393441 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkpk\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-kube-api-access-wjkpk\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.393545 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.393501 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-tls\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494220 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494193 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-trusted-ca\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494220 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-bound-sa-token\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-installation-pull-secrets\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494257 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd24f7c5-67b5-49fe-8691-06b90349e21d-ca-trust-extracted\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494272 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkpk\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-kube-api-access-wjkpk\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-tls\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494546 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494487 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-image-registry-private-configuration\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494546 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494526 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-certificates\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.494705 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.494683 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd24f7c5-67b5-49fe-8691-06b90349e21d-ca-trust-extracted\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.495290 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.495269 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-certificates\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.495427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.495403 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd24f7c5-67b5-49fe-8691-06b90349e21d-trusted-ca\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.497156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.497136 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-installation-pull-secrets\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.497284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.497259 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd24f7c5-67b5-49fe-8691-06b90349e21d-image-registry-private-configuration\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.497345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.497334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-registry-tls\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.521330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.521308 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-bound-sa-token\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.521542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.521520 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkpk\" (UniqueName: \"kubernetes.io/projected/dd24f7c5-67b5-49fe-8691-06b90349e21d-kube-api-access-wjkpk\") pod \"image-registry-56c44567bf-hdhbj\" (UID: \"dd24f7c5-67b5-49fe-8691-06b90349e21d\") " pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.622447 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.622395 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:33.944090 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:33.944045 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c44567bf-hdhbj"] Oct 29 21:42:33.946406 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:33.946380 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd24f7c5_67b5_49fe_8691_06b90349e21d.slice/crio-a2f8ac500dcda4fcf917614752492e9ab160561573f280e988ac838beda737df WatchSource:0}: Error finding container a2f8ac500dcda4fcf917614752492e9ab160561573f280e988ac838beda737df: Status 404 returned error can't find the container with id a2f8ac500dcda4fcf917614752492e9ab160561573f280e988ac838beda737df Oct 29 21:42:34.457494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:34.457466 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p75bp" Oct 29 21:42:34.486678 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:34.486641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" event={"ID":"dd24f7c5-67b5-49fe-8691-06b90349e21d","Type":"ContainerStarted","Data":"70b0720f57b3939ed656a56c6d5d87d770c92f63c0821183250fc4bd0dc24f1c"} Oct 29 21:42:34.486820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:34.486736 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:34.486820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:34.486760 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" event={"ID":"dd24f7c5-67b5-49fe-8691-06b90349e21d","Type":"ContainerStarted","Data":"a2f8ac500dcda4fcf917614752492e9ab160561573f280e988ac838beda737df"} Oct 29 21:42:34.548219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:34.548169 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" podStartSLOduration=1.548152786 podStartE2EDuration="1.548152786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:34.547817853 +0000 UTC m=+16.289748527" watchObservedRunningTime="2026-10-29 21:42:34.548152786 +0000 UTC m=+16.290083456" Oct 29 21:42:42.067701 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.067670 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2wj5m"] Oct 29 21:42:42.072308 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.072284 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.090949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.090925 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 21:42:42.091054 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.091012 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 21:42:42.102871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.102852 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 21:42:42.102967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.102910 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 21:42:42.147166 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.147148 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 21:42:42.147260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.147170 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 21:42:42.147260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.147172 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dwjqp\"" Oct 29 21:42:42.248807 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.248782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-root\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.248913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.248818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.248913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.248836 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.248913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.248855 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-textfile\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.249108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.249025 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswhd\" (UniqueName: \"kubernetes.io/projected/41636e36-7a36-4e98-920e-8a1126f6ccea-kube-api-access-bswhd\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.249108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.249094 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-tls\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.249199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.249151 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-sys\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.249199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.249169 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-metrics-client-ca\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.249199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.249188 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-wtmp\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349700 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-wtmp\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-root\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349781 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349809 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349830 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-root\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349840 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-textfile\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349890 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswhd\" (UniqueName: \"kubernetes.io/projected/41636e36-7a36-4e98-920e-8a1126f6ccea-kube-api-access-bswhd\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-wtmp\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.349928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349922 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-tls\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349972 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-sys\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.349999 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-metrics-client-ca\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.350263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41636e36-7a36-4e98-920e-8a1126f6ccea-sys\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.350282 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-textfile\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350547 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.350525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.350608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.350557 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41636e36-7a36-4e98-920e-8a1126f6ccea-metrics-client-ca\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.352338 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.352315 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.352454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.352436 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41636e36-7a36-4e98-920e-8a1126f6ccea-node-exporter-tls\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.374340 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.374314 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswhd\" (UniqueName: \"kubernetes.io/projected/41636e36-7a36-4e98-920e-8a1126f6ccea-kube-api-access-bswhd\") pod \"node-exporter-2wj5m\" (UID: \"41636e36-7a36-4e98-920e-8a1126f6ccea\") " pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.387185 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.387167 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wj5m" Oct 29 21:42:42.397271 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:42.397249 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41636e36_7a36_4e98_920e_8a1126f6ccea.slice/crio-c82662866c307d52d0b5aa6f26466f1a186b87d0ecfd98cd422b9bc2a4f9f9fa WatchSource:0}: Error finding container c82662866c307d52d0b5aa6f26466f1a186b87d0ecfd98cd422b9bc2a4f9f9fa: Status 404 returned error can't find the container with id c82662866c307d52d0b5aa6f26466f1a186b87d0ecfd98cd422b9bc2a4f9f9fa Oct 29 21:42:42.511252 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.511223 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wj5m" event={"ID":"41636e36-7a36-4e98-920e-8a1126f6ccea","Type":"ContainerStarted","Data":"c82662866c307d52d0b5aa6f26466f1a186b87d0ecfd98cd422b9bc2a4f9f9fa"} Oct 29 21:42:42.974759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:42.974731 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-znvhf" Oct 29 21:42:43.515500 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:43.515444 4543 generic.go:356] "Generic (PLEG): container finished" podID="41636e36-7a36-4e98-920e-8a1126f6ccea" containerID="5702d04064bd997fa1b1d52faa6233793d8e80a7b00714a0a59ed67a3715c5fa" exitCode=0 Oct 29 21:42:43.515780 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:43.515515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wj5m" event={"ID":"41636e36-7a36-4e98-920e-8a1126f6ccea","Type":"ContainerDied","Data":"5702d04064bd997fa1b1d52faa6233793d8e80a7b00714a0a59ed67a3715c5fa"} Oct 29 21:42:44.244773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.244737 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f85cdbb4f-87l69"] Oct 29 21:42:44.248425 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.248407 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.279632 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.279602 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 21:42:44.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.279924 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 21:42:44.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.279948 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f2rbrrn6et0ev\"" Oct 29 21:42:44.280156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.279998 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q75vx\"" Oct 29 21:42:44.280156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.280003 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 21:42:44.280156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.280066 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 21:42:44.280303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.280160 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 21:42:44.281182 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.281161 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f85cdbb4f-87l69"] Oct 29 21:42:44.363069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363043 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b24797-fe50-4957-a874-de6bf3feb83a-metrics-client-ca\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363101 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363147 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzrw\" (UniqueName: \"kubernetes.io/projected/68b24797-fe50-4957-a874-de6bf3feb83a-kube-api-access-pnzrw\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363171 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363310 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363197 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363310 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363304 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-grpc-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363326 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68b24797-fe50-4957-a874-de6bf3feb83a-tmp\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.363382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.363346 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.463796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463771 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.463891 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzrw\" (UniqueName: \"kubernetes.io/projected/68b24797-fe50-4957-a874-de6bf3feb83a-kube-api-access-pnzrw\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.463891 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463867 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.463981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463915 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.463981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463949 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.464110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.463983 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-grpc-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.464110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.464013 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68b24797-fe50-4957-a874-de6bf3feb83a-tmp\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.464110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.464051 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.464110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.464104 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b24797-fe50-4957-a874-de6bf3feb83a-metrics-client-ca\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.464540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.464504 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68b24797-fe50-4957-a874-de6bf3feb83a-tmp\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.465282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.465257 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b24797-fe50-4957-a874-de6bf3feb83a-metrics-client-ca\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.466918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.466889 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.466918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.466908 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.467316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.467286 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.467466 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.467441 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-grpc-tls\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.467624 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.467603 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.467741 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.467720 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b24797-fe50-4957-a874-de6bf3feb83a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.488300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.488279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzrw\" (UniqueName: \"kubernetes.io/projected/68b24797-fe50-4957-a874-de6bf3feb83a-kube-api-access-pnzrw\") pod \"thanos-querier-7f85cdbb4f-87l69\" (UID: \"68b24797-fe50-4957-a874-de6bf3feb83a\") " pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.520751 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.520692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wj5m" event={"ID":"41636e36-7a36-4e98-920e-8a1126f6ccea","Type":"ContainerStarted","Data":"ffcbc212ef14e0fc02532a837ada441c6840d52f108e2feebb9e5bf708a82419"} Oct 29 21:42:44.520751 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.520737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wj5m" event={"ID":"41636e36-7a36-4e98-920e-8a1126f6ccea","Type":"ContainerStarted","Data":"8598fd1ee763053d7b020608c9294db50f8ffff118fc7d6ac2796a1ee79bbb92"} Oct 29 21:42:44.565215 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.565168 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2wj5m" podStartSLOduration=1.651806069 podStartE2EDuration="2.565157651s" totalImagesPullingTime="840.118065ms" totalInitContainerRuntime="73.233517ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:44.564398209 +0000 UTC m=+26.306328881" watchObservedRunningTime="2026-10-29 21:42:44.565157651 +0000 UTC m=+26.307088323" Oct 29 21:42:44.570681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.570651 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:44.914808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:44.914765 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f85cdbb4f-87l69"] Oct 29 21:42:45.525824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:45.525788 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"d8ec2b6bd31a803734e76db28caf03efd6e7639f2aec23faae4cca2dd69c65cc"} Oct 29 21:42:47.001226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:47.001202 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2thvl" Oct 29 21:42:47.534233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:47.534200 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"1236761a80e311635bf43fe129104587f3058ad24192366c80a7f42185259ecf"} Oct 29 21:42:47.534378 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:47.534265 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"8fdb056873b0276893c8f417ed06a3d9ee86b8f53c8c95389b1f81c6b64812b5"} Oct 29 21:42:47.534378 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:47.534293 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"135c1fae875673b0569c37d158630605196e8d4262c6312bc9e79ee90ec79b5e"} Oct 29 21:42:48.539580 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:48.539548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"6d10f482660aa07050d8a3d75aa055ef35a5d8ea2b071f975feed31219957bea"} Oct 29 21:42:48.540006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:48.539628 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"274f99d77ddfbb335395d3d232873ece84c8cd9fbdbfc682e9028d356247f1f4"} Oct 29 21:42:48.540006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:48.539654 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" event={"ID":"68b24797-fe50-4957-a874-de6bf3feb83a","Type":"ContainerStarted","Data":"9003b48654da9e4e9add56093bcb45b6e1e8c6eb721096f58bfc5d3b1f96e581"} Oct 29 21:42:48.540006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:48.539732 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:48.619015 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:48.618972 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" podStartSLOduration=1.887905481 podStartE2EDuration="4.618960429s" totalImagesPullingTime="2.731054948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:48.60673844 +0000 UTC m=+30.348669112" watchObservedRunningTime="2026-10-29 21:42:48.618960429 +0000 UTC m=+30.360891080" Oct 29 21:42:49.185897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.185860 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:42:49.206366 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.206337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.223347 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.223315 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 21:42:49.223456 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.223422 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 21:42:49.223520 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.223480 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 21:42:49.223981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.223964 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 21:42:49.224243 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.224226 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 21:42:49.236125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.236046 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 21:42:49.236125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.236116 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 21:42:49.236321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.236146 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lxq9t\"" Oct 29 21:42:49.236321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.236176 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 21:42:49.236321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.236229 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 21:42:49.237716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.237698 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:42:49.298774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298749 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.298857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298776 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.298857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298793 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.298857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298820 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.298984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298882 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.298984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.298993 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dnd\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299107 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299173 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299216 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299242 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299306 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299324 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299348 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299371 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299391 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.299551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.299414 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.323215 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.323177 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 21:42:49.323215 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.323197 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 21:42:49.323345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.323331 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edvk94qhnijaq\"" Oct 29 21:42:49.324183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.324169 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 21:42:49.324264 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.324230 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 21:42:49.337469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.337438 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 21:42:49.337561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.337518 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 21:42:49.400345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400318 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400349 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400375 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400560 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400495 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400560 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400526 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400556 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400588 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400624 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400649 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400729 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400773 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400808 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400832 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.400864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400856 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400886 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400916 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400952 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.400998 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401029 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dnd\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401057 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401117 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401505 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401532 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.401680 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.401662 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.402249 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.402219 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.402482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.402455 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.403382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.403354 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.404561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.404536 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.405312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405262 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.405898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405622 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.405898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405626 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.405898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405810 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.405898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405827 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.406239 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.405987 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.406239 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.406202 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.406597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.406559 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.406872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.406850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.407000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.406947 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.408254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.408232 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.426657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.426633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dnd\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\") pod \"prometheus-k8s-0\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.611163 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.611135 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:42:49.980249 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:49.980225 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:42:49.982650 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:42:49.982618 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a83115_1d36_48ba_a49d_6ab14412c813.slice/crio-d6097bd0743a50c1ce5ff9dd15ff02ac9fbe94d268878b470a098335230158f8 WatchSource:0}: Error finding container d6097bd0743a50c1ce5ff9dd15ff02ac9fbe94d268878b470a098335230158f8: Status 404 returned error can't find the container with id d6097bd0743a50c1ce5ff9dd15ff02ac9fbe94d268878b470a098335230158f8 Oct 29 21:42:50.547273 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:50.547245 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"d6097bd0743a50c1ce5ff9dd15ff02ac9fbe94d268878b470a098335230158f8"} Oct 29 21:42:51.552730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:51.552704 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a"} Oct 29 21:42:52.558403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:52.558364 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a" exitCode=0 Oct 29 21:42:52.558826 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:52.558457 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a"} Oct 29 21:42:54.548692 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:54.548663 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f85cdbb4f-87l69" Oct 29 21:42:55.495192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:55.495173 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56c44567bf-hdhbj" Oct 29 21:42:55.575238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:55.574193 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f"} Oct 29 21:42:55.575238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:55.574278 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89"} Oct 29 21:42:55.575238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:55.574318 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2"} Oct 29 21:42:56.580532 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:56.580497 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f"} Oct 29 21:42:56.580967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:56.580569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a"} Oct 29 21:42:56.581051 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:56.581006 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerStarted","Data":"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f"} Oct 29 21:42:56.652537 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:56.651906 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.344675929 podStartE2EDuration="7.651889523s" totalImagesPullingTime="4.285993398s" totalInitContainerRuntime="21.220196ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:56.650299093 +0000 UTC m=+38.392229767" watchObservedRunningTime="2026-10-29 21:42:56.651889523 +0000 UTC m=+38.393820197" Oct 29 21:42:57.959183 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:42:57.959154 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:42:59.611686 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:42:59.611605 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:26.540451 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.540420 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:43:26.540979 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.540683 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" podUID="0b49a2a6-8054-4068-bd32-fdd07397691a" containerName="insights-gathering" containerID="cri-o://eeeccef5255bcb381a40028ec250779bcd4021a38e07f03142335a0b5c792b34" gracePeriod=30 Oct 29 21:43:26.698229 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.698204 4543 generic.go:356] "Generic (PLEG): container finished" podID="0b49a2a6-8054-4068-bd32-fdd07397691a" containerID="eeeccef5255bcb381a40028ec250779bcd4021a38e07f03142335a0b5c792b34" exitCode=2 Oct 29 21:43:26.698323 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.698241 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" event={"ID":"0b49a2a6-8054-4068-bd32-fdd07397691a","Type":"ContainerDied","Data":"eeeccef5255bcb381a40028ec250779bcd4021a38e07f03142335a0b5c792b34"} Oct 29 21:43:26.833857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.833831 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.874981 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") pod \"0b49a2a6-8054-4068-bd32-fdd07397691a\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875029 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") pod \"0b49a2a6-8054-4068-bd32-fdd07397691a\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875100 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") pod \"0b49a2a6-8054-4068-bd32-fdd07397691a\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") pod \"0b49a2a6-8054-4068-bd32-fdd07397691a\" (UID: \"0b49a2a6-8054-4068-bd32-fdd07397691a\") " Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875353 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle" pod "0b49a2a6-8054-4068-bd32-fdd07397691a" (UID: "0b49a2a6-8054-4068-bd32-fdd07397691a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:26.875457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875421 4543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-trusted-ca-bundle\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:26.875845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path" pod "0b49a2a6-8054-4068-bd32-fdd07397691a" (UID: "0b49a2a6-8054-4068-bd32-fdd07397691a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:26.875845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.875688 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle" pod "0b49a2a6-8054-4068-bd32-fdd07397691a" (UID: "0b49a2a6-8054-4068-bd32-fdd07397691a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:26.878053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.878029 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl" pod "0b49a2a6-8054-4068-bd32-fdd07397691a" (UID: "0b49a2a6-8054-4068-bd32-fdd07397691a"). InnerVolumeSpecName "kube-api-access-557nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:26.976388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.976366 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-557nl\" (UniqueName: \"kubernetes.io/projected/0b49a2a6-8054-4068-bd32-fdd07397691a-kube-api-access-557nl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:26.976388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.976387 4543 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b49a2a6-8054-4068-bd32-fdd07397691a-service-ca-bundle\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:26.976523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:26.976397 4543 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b49a2a6-8054-4068-bd32-fdd07397691a-archives-path\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:27.707705 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:27.707666 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" event={"ID":"0b49a2a6-8054-4068-bd32-fdd07397691a","Type":"ContainerDied","Data":"f008998401fde0e0158bb8f7b1bb1a6217968cbb197a6265b0d7bd3191f1af8e"} Oct 29 21:43:27.708155 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:27.707723 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8vksd-8q4gh" Oct 29 21:43:27.708155 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:27.707755 4543 scope.go:122] "RemoveContainer" containerID="eeeccef5255bcb381a40028ec250779bcd4021a38e07f03142335a0b5c792b34" Oct 29 21:43:27.760367 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:27.760341 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:43:27.772665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:27.772636 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-8vksd-8q4gh"] Oct 29 21:43:27.962485 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:27.962452 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:43:28.382347 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:28.382280 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b49a2a6-8054-4068-bd32-fdd07397691a" path="/var/lib/kubelet/pods/0b49a2a6-8054-4068-bd32-fdd07397691a/volumes" Oct 29 21:43:40.734092 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:40.734037 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:40.756714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:40.756686 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:40.772895 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:40.772873 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:57.960205 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:57.960168 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:43:58.487295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487265 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:43:58.487589 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487552 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="prometheus" containerID="cri-o://2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2" gracePeriod=600 Oct 29 21:43:58.487758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487578 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="thanos-sidecar" containerID="cri-o://0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f" gracePeriod=600 Oct 29 21:43:58.487758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487591 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy" containerID="cri-o://bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a" gracePeriod=600 Oct 29 21:43:58.487758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487627 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="config-reloader" containerID="cri-o://2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89" gracePeriod=600 Oct 29 21:43:58.487758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487626 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-web" containerID="cri-o://74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f" gracePeriod=600 Oct 29 21:43:58.487758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.487656 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f" gracePeriod=600 Oct 29 21:43:58.818066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818011 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f" exitCode=0 Oct 29 21:43:58.818066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818035 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a" exitCode=0 Oct 29 21:43:58.818066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818041 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f" exitCode=0 Oct 29 21:43:58.818066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818047 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89" exitCode=0 Oct 29 21:43:58.818066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818052 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2" exitCode=0 Oct 29 21:43:58.818301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818094 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f"} Oct 29 21:43:58.818301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818153 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a"} Oct 29 21:43:58.818301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818182 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f"} Oct 29 21:43:58.818301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818208 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89"} Oct 29 21:43:58.818301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:58.818232 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2"} Oct 29 21:43:59.738130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.738109 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:59.823873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.823811 4543 generic.go:356] "Generic (PLEG): container finished" podID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerID="74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f" exitCode=0 Oct 29 21:43:59.823873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.823854 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f"} Oct 29 21:43:59.824022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.823912 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a83115-1d36-48ba-a49d-6ab14412c813","Type":"ContainerDied","Data":"d6097bd0743a50c1ce5ff9dd15ff02ac9fbe94d268878b470a098335230158f8"} Oct 29 21:43:59.824022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.823943 4543 scope.go:122] "RemoveContainer" containerID="6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f" Oct 29 21:43:59.824022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.823957 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:43:59.831483 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.831463 4543 scope.go:122] "RemoveContainer" containerID="bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a" Oct 29 21:43:59.840675 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.840650 4543 scope.go:122] "RemoveContainer" containerID="74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f" Oct 29 21:43:59.848473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.848453 4543 scope.go:122] "RemoveContainer" containerID="0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f" Oct 29 21:43:59.855377 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.855362 4543 scope.go:122] "RemoveContainer" containerID="2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89" Oct 29 21:43:59.862377 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.862359 4543 scope.go:122] "RemoveContainer" containerID="2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2" Oct 29 21:43:59.869963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.869941 4543 scope.go:122] "RemoveContainer" containerID="07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a" Oct 29 21:43:59.877478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.877464 4543 scope.go:122] "RemoveContainer" containerID="6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f" Oct 29 21:43:59.877749 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.877729 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f\": container with ID starting with 6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f not found: ID does not exist" containerID="6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f" Oct 29 21:43:59.877813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.877759 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f"} err="failed to get container status \"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f\": rpc error: code = NotFound desc = could not find container \"6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f\": container with ID starting with 6914c42235017770dc928f7941f526825aacc43cec8f14b441ef97032381cb5f not found: ID does not exist" Oct 29 21:43:59.877813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.877791 4543 scope.go:122] "RemoveContainer" containerID="bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a" Oct 29 21:43:59.878048 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.878026 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a\": container with ID starting with bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a not found: ID does not exist" containerID="bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a" Oct 29 21:43:59.878118 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878060 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a"} err="failed to get container status \"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a\": rpc error: code = NotFound desc = could not find container \"bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a\": container with ID starting with bf887e2beb47b7db157011a83cdc6daacf5781b08d7c637a1eaa74b574fdd00a not found: ID does not exist" Oct 29 21:43:59.878118 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878106 4543 scope.go:122] "RemoveContainer" containerID="74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f" Oct 29 21:43:59.878369 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.878353 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f\": container with ID starting with 74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f not found: ID does not exist" containerID="74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f" Oct 29 21:43:59.878430 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878372 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f"} err="failed to get container status \"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f\": rpc error: code = NotFound desc = could not find container \"74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f\": container with ID starting with 74aabde6077859a62b7dad1b03af8c547d8f543e35da206c6d69698d3a2c728f not found: ID does not exist" Oct 29 21:43:59.878430 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878385 4543 scope.go:122] "RemoveContainer" containerID="0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f" Oct 29 21:43:59.878626 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.878610 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f\": container with ID starting with 0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f not found: ID does not exist" containerID="0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f" Oct 29 21:43:59.878676 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878628 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f"} err="failed to get container status \"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f\": rpc error: code = NotFound desc = could not find container \"0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f\": container with ID starting with 0e6cd67b3527d50629028fb91c0d73ecb7e345bfff61ccf730500399fc888c9f not found: ID does not exist" Oct 29 21:43:59.878676 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878638 4543 scope.go:122] "RemoveContainer" containerID="2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89" Oct 29 21:43:59.878862 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.878846 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89\": container with ID starting with 2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89 not found: ID does not exist" containerID="2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89" Oct 29 21:43:59.878903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878881 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89"} err="failed to get container status \"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89\": rpc error: code = NotFound desc = could not find container \"2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89\": container with ID starting with 2f48a38a2640aa7fd457526ac3e8ed0c3f88ff364c18aa2c1947b96d06588b89 not found: ID does not exist" Oct 29 21:43:59.878934 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.878902 4543 scope.go:122] "RemoveContainer" containerID="2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2" Oct 29 21:43:59.879211 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.879195 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2\": container with ID starting with 2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2 not found: ID does not exist" containerID="2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2" Oct 29 21:43:59.879280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.879211 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2"} err="failed to get container status \"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2\": rpc error: code = NotFound desc = could not find container \"2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2\": container with ID starting with 2e696efe240804022be9408ebfbedcd722b5fb8b8bf3028300be53985d91e9e2 not found: ID does not exist" Oct 29 21:43:59.879280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.879225 4543 scope.go:122] "RemoveContainer" containerID="07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a" Oct 29 21:43:59.879460 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:43:59.879446 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a\": container with ID starting with 07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a not found: ID does not exist" containerID="07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a" Oct 29 21:43:59.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.879462 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a"} err="failed to get container status \"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a\": rpc error: code = NotFound desc = could not find container \"07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a\": container with ID starting with 07e54ffbdc7f8d666975e4093bf1a626fd2fa5f5ff48e574c97f7a2323041e7a not found: ID does not exist" Oct 29 21:43:59.897719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897699 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.897781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897726 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.897781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.897781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897766 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.897922 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897904 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.897996 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.897977 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898052 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898024 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898060 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898059 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.898125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898107 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898122 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.898277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898137 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898168 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898205 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898237 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898293 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898324 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898358 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898414 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898450 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898483 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898532 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\") pod \"a1a83115-1d36-48ba-a49d-6ab14412c813\" (UID: \"a1a83115-1d36-48ba-a49d-6ab14412c813\") " Oct 29 21:43:59.898782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898735 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.898782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898753 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.898782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.898726 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.899506 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.899410 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.899920 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.899694 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.899920 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.899826 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:59.900065 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.900028 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.900218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.900197 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:59.901260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.901238 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:59.901703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.901656 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.901703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.901693 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.902680 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.902643 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.902784 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.902735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.902862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.902830 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.902995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.902972 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "kube-api-access-m7dnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:59.903140 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.903119 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:59.903443 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.903415 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.904295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.904280 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.904639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.904622 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.913068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.913044 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config" pod "a1a83115-1d36-48ba-a49d-6ab14412c813" (UID: "a1a83115-1d36-48ba-a49d-6ab14412c813"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:59.999469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999447 4543 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-kube-rbac-proxy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999468 4543 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-config\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999478 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999489 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999500 4543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-web-config\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999508 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999516 4543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-config-out\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999525 4543 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-metrics-client-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999533 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999541 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999550 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999559 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a83115-1d36-48ba-a49d-6ab14412c813-prometheus-k8s-db\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999567 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7dnd\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-kube-api-access-m7dnd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999575 4543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a83115-1d36-48ba-a49d-6ab14412c813-tls-assets\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999583 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a83115-1d36-48ba-a49d-6ab14412c813-configmap-metrics-client-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999592 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999600 4543 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-secret-grpc-tls\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:43:59.999889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:43:59.999608 4543 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a83115-1d36-48ba-a49d-6ab14412c813-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:44:00.177725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.174143 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:44:00.195183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.195163 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:44:00.236292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236274 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:44:00.236519 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236506 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="init-config-reloader" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236521 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="init-config-reloader" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236536 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="config-reloader" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236541 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="config-reloader" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236552 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0b49a2a6-8054-4068-bd32-fdd07397691a" containerName="insights-gathering" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236557 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b49a2a6-8054-4068-bd32-fdd07397691a" containerName="insights-gathering" Oct 29 21:44:00.236565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236565 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236569 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236576 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-web" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236580 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-web" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236586 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-thanos" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236590 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-thanos" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236597 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="prometheus" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236601 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="prometheus" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236608 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="thanos-sidecar" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236611 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="thanos-sidecar" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236648 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-web" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236655 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy-thanos" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236662 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="prometheus" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236668 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="config-reloader" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236674 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="thanos-sidecar" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236680 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b49a2a6-8054-4068-bd32-fdd07397691a" containerName="insights-gathering" Oct 29 21:44:00.236767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.236686 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" containerName="kube-rbac-proxy" Oct 29 21:44:00.240601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.240587 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.270894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270671 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 21:44:00.270894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270700 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 21:44:00.270894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270707 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 21:44:00.270894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270849 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 21:44:00.271204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270921 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 21:44:00.271204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.270981 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 21:44:00.271712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.271693 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lxq9t\"" Oct 29 21:44:00.272218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.271841 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 21:44:00.272330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.271892 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 21:44:00.272330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.271947 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 21:44:00.272454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.271977 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 21:44:00.272454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.272020 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 21:44:00.272454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.272047 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 21:44:00.272454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.272101 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edvk94qhnijaq\"" Oct 29 21:44:00.272454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.272128 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 21:44:00.284745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.284721 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 21:44:00.284745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.284721 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 21:44:00.285142 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.285124 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:44:00.382204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.382156 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a83115-1d36-48ba-a49d-6ab14412c813" path="/var/lib/kubelet/pods/a1a83115-1d36-48ba-a49d-6ab14412c813/volumes" Oct 29 21:44:00.402698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402705 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402722 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config-out\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402736 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402870 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402870 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402846 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402870 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402866 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.402963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402933 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403003 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402966 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403003 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.402985 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403017 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403055 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-web-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403091 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403109 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403135 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403164 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403192 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzpbq\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-kube-api-access-mzpbq\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403216 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403234 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.403277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.403252 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504310 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config-out\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504360 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504379 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504404 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504633 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504695 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504660 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504695 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504690 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504715 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504742 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504768 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-web-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504805 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504831 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504863 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504886 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504913 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzpbq\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-kube-api-access-mzpbq\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.504941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504939 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.505286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.504973 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.505286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.505003 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.505286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.505241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.505993 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.505477 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.505993 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.505936 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.507484 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.507450 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config-out\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.507610 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.507587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.507797 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.507817 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508122 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508327 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508719 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508813 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508927 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.508991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.508962 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-web-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.509488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.509380 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.509595 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.509568 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.509857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.509835 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.510896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.510871 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-config\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.511027 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.511011 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.511127 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.511108 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.533310 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.533293 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzpbq\" (UniqueName: \"kubernetes.io/projected/0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba-kube-api-access-mzpbq\") pod \"prometheus-k8s-0\" (UID: \"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.556897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.556876 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:00.887480 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:00.887455 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 21:44:00.888877 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:44:00.888848 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0297c7a4_dfb0_4b05_a7cd_2d77f9a6c3ba.slice/crio-a21ee93ebe4883175d3473ba75d2af2d929bcdde0d6afa2c18a50bfa1dfd24a2 WatchSource:0}: Error finding container a21ee93ebe4883175d3473ba75d2af2d929bcdde0d6afa2c18a50bfa1dfd24a2: Status 404 returned error can't find the container with id a21ee93ebe4883175d3473ba75d2af2d929bcdde0d6afa2c18a50bfa1dfd24a2 Oct 29 21:44:01.833714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:01.833678 4543 generic.go:356] "Generic (PLEG): container finished" podID="0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba" containerID="761016436e4e45c2560c0b83cbf04f52d020f6bfbe4f92596bacadc74a21bac9" exitCode=0 Oct 29 21:44:01.833894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:01.833771 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerDied","Data":"761016436e4e45c2560c0b83cbf04f52d020f6bfbe4f92596bacadc74a21bac9"} Oct 29 21:44:01.833894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:01.833877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"a21ee93ebe4883175d3473ba75d2af2d929bcdde0d6afa2c18a50bfa1dfd24a2"} Oct 29 21:44:02.840953 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.840912 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"13c7832ca94686531e0aedb579542bff2c848149e9bd7687f30adfc91485f201"} Oct 29 21:44:02.841410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.840988 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"280567b77c215b840d89e2ca097c5e95e64a1586d1b088e7b96406e84d107e10"} Oct 29 21:44:02.841410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.841029 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"81684f4c02a186643445451189000fd516fec0eb2120952a1bf710f6850987c4"} Oct 29 21:44:02.841410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.841056 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"b62daa1a364da643941a5c909e1607588db88d4a211d688ce69ef4dfbd2647d3"} Oct 29 21:44:02.841410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.841106 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"cae23875014be2b034ed9787b4e2a796632e99e7a9c5a7adc2afdfaa81b020ed"} Oct 29 21:44:02.841410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.841135 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba","Type":"ContainerStarted","Data":"dca72026c21308963e28d2979cc6342cfd614555c7ee9bdaf8b7747a65e35a7b"} Oct 29 21:44:02.926338 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:02.926281 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.908580651 podStartE2EDuration="2.92626137s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.680719ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:44:02.925144894 +0000 UTC m=+104.667075567" watchObservedRunningTime="2026-10-29 21:44:02.92626137 +0000 UTC m=+104.668192044" Oct 29 21:44:05.557918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:44:05.557879 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:44:27.960687 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:44:27.960646 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:44:57.961932 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:44:57.961849 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:45:00.557710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:45:00.557671 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:45:00.575294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:45:00.575254 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:45:01.055235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:45:01.055210 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 21:45:27.965205 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:45:27.965163 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:45:57.962683 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:45:57.962642 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:46:27.962067 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:46:27.962021 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:46:57.962258 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:46:57.962212 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:47:27.962012 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:47:27.961965 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:47:32.009682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.009594 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb"] Oct 29 21:47:32.012861 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.012828 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.035602 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.035573 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgm7\" (UniqueName: \"kubernetes.io/projected/073baece-bb45-4169-afba-fe80d608df01-kube-api-access-lxgm7\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.035761 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.035634 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073baece-bb45-4169-afba-fe80d608df01-tmp\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.040637 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.040612 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-9292r\"" Oct 29 21:47:32.040637 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.040635 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 29 21:47:32.040846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.040661 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:47:32.041694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.041673 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb"] Oct 29 21:47:32.137050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.137016 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgm7\" (UniqueName: \"kubernetes.io/projected/073baece-bb45-4169-afba-fe80d608df01-kube-api-access-lxgm7\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.137199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.137066 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073baece-bb45-4169-afba-fe80d608df01-tmp\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.137427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.137411 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073baece-bb45-4169-afba-fe80d608df01-tmp\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.162140 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.162114 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgm7\" (UniqueName: \"kubernetes.io/projected/073baece-bb45-4169-afba-fe80d608df01-kube-api-access-lxgm7\") pod \"jobset-operator-674cd59dc5-spjhb\" (UID: \"073baece-bb45-4169-afba-fe80d608df01\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.342832 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.342775 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" Oct 29 21:47:32.644140 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.644059 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb"] Oct 29 21:47:32.648586 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:47:32.648557 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod073baece_bb45_4169_afba_fe80d608df01.slice/crio-ec371357bcda28c4c9079b365d21f51913cb7ffc14d35112ccb1223f8a263d14 WatchSource:0}: Error finding container ec371357bcda28c4c9079b365d21f51913cb7ffc14d35112ccb1223f8a263d14: Status 404 returned error can't find the container with id ec371357bcda28c4c9079b365d21f51913cb7ffc14d35112ccb1223f8a263d14 Oct 29 21:47:32.650064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:32.650042 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:47:33.559589 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:33.559514 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" event={"ID":"073baece-bb45-4169-afba-fe80d608df01","Type":"ContainerStarted","Data":"ec371357bcda28c4c9079b365d21f51913cb7ffc14d35112ccb1223f8a263d14"} Oct 29 21:47:35.569608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:35.569575 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" event={"ID":"073baece-bb45-4169-afba-fe80d608df01","Type":"ContainerStarted","Data":"95073e0aef7e01019ea054ebdef700bdac1af4ef7e89b93fd722e619fb018392"} Oct 29 21:47:35.612547 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:35.612504 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-spjhb" podStartSLOduration=2.091136306 podStartE2EDuration="4.612491947s" totalImagesPullingTime="2.521355641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:35.611977891 +0000 UTC m=+317.353908563" watchObservedRunningTime="2026-10-29 21:47:35.612491947 +0000 UTC m=+317.354422621" Oct 29 21:47:50.666930 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.666895 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n"] Oct 29 21:47:50.673417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.673396 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.696491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.696471 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n"] Oct 29 21:47:50.741226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741205 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 21:47:50.741520 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741499 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 21:47:50.741765 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741741 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-flksv\"" Oct 29 21:47:50.741864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741771 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 21:47:50.741864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741778 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 21:47:50.741972 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.741955 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 21:47:50.766860 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.766840 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.766981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.766867 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.766981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.766888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.766981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.766912 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.766981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.766967 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqj55\" (UniqueName: \"kubernetes.io/projected/67166238-9a30-49da-9b04-efa7b2658cce-kube-api-access-lqj55\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.767142 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.767090 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.867914 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.867890 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.867917 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.867936 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.867959 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.867974 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj55\" (UniqueName: \"kubernetes.io/projected/67166238-9a30-49da-9b04-efa7b2658cce-kube-api-access-lqj55\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868244 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.868115 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.868495 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.868465 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.870615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.870583 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67166238-9a30-49da-9b04-efa7b2658cce-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.870730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.870621 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.870794 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.870754 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.870924 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.870900 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67166238-9a30-49da-9b04-efa7b2658cce-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.894126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.894096 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj55\" (UniqueName: \"kubernetes.io/projected/67166238-9a30-49da-9b04-efa7b2658cce-kube-api-access-lqj55\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-qjv5n\" (UID: \"67166238-9a30-49da-9b04-efa7b2658cce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:50.989184 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:50.989158 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:51.355578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.355489 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n"] Oct 29 21:47:51.360004 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:47:51.359959 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67166238_9a30_49da_9b04_efa7b2658cce.slice/crio-181927d2431f1d0981cd2d24935c88a890ffd419eb26e6af50f387073a48e9c9 WatchSource:0}: Error finding container 181927d2431f1d0981cd2d24935c88a890ffd419eb26e6af50f387073a48e9c9: Status 404 returned error can't find the container with id 181927d2431f1d0981cd2d24935c88a890ffd419eb26e6af50f387073a48e9c9 Oct 29 21:47:51.624848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.624777 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" event={"ID":"67166238-9a30-49da-9b04-efa7b2658cce","Type":"ContainerStarted","Data":"181927d2431f1d0981cd2d24935c88a890ffd419eb26e6af50f387073a48e9c9"} Oct 29 21:47:51.968408 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.968371 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf"] Oct 29 21:47:51.972312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.971946 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:51.992132 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.992111 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 29 21:47:51.994264 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.994237 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 29 21:47:51.995926 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:51.995907 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf"] Oct 29 21:47:52.021197 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.021174 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-96qg5\"" Oct 29 21:47:52.077729 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.077696 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.077855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.077776 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4twv\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-kube-api-access-h4twv\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.178429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.178396 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4twv\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-kube-api-access-h4twv\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.178556 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.178470 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.213431 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.213407 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4twv\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-kube-api-access-h4twv\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.220749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.220698 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/543853b5-9761-4852-8801-1519786ced31-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4rnjf\" (UID: \"543853b5-9761-4852-8801-1519786ced31\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.285858 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.285829 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:52.602685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.602615 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf"] Oct 29 21:47:52.606959 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:47:52.606921 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543853b5_9761_4852_8801_1519786ced31.slice/crio-0115f2d622f9a526669c1aff77775cc369a0ab399df57565660f592365f2bbc2 WatchSource:0}: Error finding container 0115f2d622f9a526669c1aff77775cc369a0ab399df57565660f592365f2bbc2: Status 404 returned error can't find the container with id 0115f2d622f9a526669c1aff77775cc369a0ab399df57565660f592365f2bbc2 Oct 29 21:47:52.634813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:52.634782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" event={"ID":"543853b5-9761-4852-8801-1519786ced31","Type":"ContainerStarted","Data":"0115f2d622f9a526669c1aff77775cc369a0ab399df57565660f592365f2bbc2"} Oct 29 21:47:53.564048 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.564012 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9"] Oct 29 21:47:53.567679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.567657 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.590327 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.590298 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9"] Oct 29 21:47:53.595575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.595554 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bmjxj\"" Oct 29 21:47:53.691447 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.691358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfzb\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-kube-api-access-cpfzb\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.691447 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.691416 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.792880 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.792837 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfzb\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-kube-api-access-cpfzb\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.793054 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.792902 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.820720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.820657 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfzb\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-kube-api-access-cpfzb\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.821732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.821709 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa527c97-9b2d-4889-86dc-cb1bca77e732-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5l2t9\" (UID: \"aa527c97-9b2d-4889-86dc-cb1bca77e732\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:53.882664 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:53.882632 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" Oct 29 21:47:54.217473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:54.217440 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9"] Oct 29 21:47:54.229268 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:47:54.229238 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa527c97_9b2d_4889_86dc_cb1bca77e732.slice/crio-b991d35534218e1e2f15d2758eb4d372c9f0fad400c6899ac7728e0cd32f7b43 WatchSource:0}: Error finding container b991d35534218e1e2f15d2758eb4d372c9f0fad400c6899ac7728e0cd32f7b43: Status 404 returned error can't find the container with id b991d35534218e1e2f15d2758eb4d372c9f0fad400c6899ac7728e0cd32f7b43 Oct 29 21:47:54.645191 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:54.645114 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" event={"ID":"aa527c97-9b2d-4889-86dc-cb1bca77e732","Type":"ContainerStarted","Data":"b991d35534218e1e2f15d2758eb4d372c9f0fad400c6899ac7728e0cd32f7b43"} Oct 29 21:47:55.653745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:55.651982 4543 generic.go:356] "Generic (PLEG): container finished" podID="67166238-9a30-49da-9b04-efa7b2658cce" containerID="ca273d785146586db78dec6e46461aafdc5a954ca8b6a8f827bd9847aade6aaa" exitCode=0 Oct 29 21:47:55.653745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:55.652097 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" event={"ID":"67166238-9a30-49da-9b04-efa7b2658cce","Type":"ContainerDied","Data":"ca273d785146586db78dec6e46461aafdc5a954ca8b6a8f827bd9847aade6aaa"} Oct 29 21:47:56.657713 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.657663 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" event={"ID":"67166238-9a30-49da-9b04-efa7b2658cce","Type":"ContainerStarted","Data":"2aafdbbe81404cb840622e687a0a27ab81c704c8bdd1582dd0c9aba1cce5da46"} Oct 29 21:47:56.658260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.657826 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:47:56.659377 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.659353 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" event={"ID":"543853b5-9761-4852-8801-1519786ced31","Type":"ContainerStarted","Data":"a77296cd486e85f10632a959fda395b3be794d8f49148f7c44ba5bfdf6ff7a7a"} Oct 29 21:47:56.659517 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.659403 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:47:56.725414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.725358 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" podStartSLOduration=2.433439624 podStartE2EDuration="6.72534603s" totalImagesPullingTime="3.692412518s" totalInitContainerRuntime="599.493888ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:56.72403444 +0000 UTC m=+338.465965112" watchObservedRunningTime="2026-10-29 21:47:56.72534603 +0000 UTC m=+338.467276719" Oct 29 21:47:56.768273 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:56.768230 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" podStartSLOduration=2.234267559 podStartE2EDuration="5.76821443s" totalImagesPullingTime="3.533946871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:56.767846948 +0000 UTC m=+338.509777623" watchObservedRunningTime="2026-10-29 21:47:56.76821443 +0000 UTC m=+338.510145107" Oct 29 21:47:57.664154 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:57.664116 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" event={"ID":"aa527c97-9b2d-4889-86dc-cb1bca77e732","Type":"ContainerStarted","Data":"d4e117fe25ed8196beff2ac2370ebced5387c08d50612eec9eb2af339ed3f809"} Oct 29 21:47:57.705124 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:47:57.705044 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5l2t9" podStartSLOduration=1.9497615430000002 podStartE2EDuration="4.704993048s" totalImagesPullingTime="2.755231505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:57.704684133 +0000 UTC m=+339.446614807" watchObservedRunningTime="2026-10-29 21:47:57.704993048 +0000 UTC m=+339.446923722" Oct 29 21:47:57.960931 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:47:57.960897 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:48:02.666578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:02.666542 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" Oct 29 21:48:03.651235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.651197 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-v7494"] Oct 29 21:48:03.656368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.656348 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.683410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.683386 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-v7494"] Oct 29 21:48:03.694540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.694514 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ncn7c\"" Oct 29 21:48:03.769936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.769907 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgq9\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-kube-api-access-zxgq9\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.770131 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.769974 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-bound-sa-token\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.870459 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.870421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgq9\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-kube-api-access-zxgq9\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.870571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.870484 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-bound-sa-token\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.897908 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.897873 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-bound-sa-token\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.898013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.897925 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgq9\" (UniqueName: \"kubernetes.io/projected/bedbe3b3-df9c-49da-834e-48d64cbda054-kube-api-access-zxgq9\") pod \"cert-manager-597b95cb84-v7494\" (UID: \"bedbe3b3-df9c-49da-834e-48d64cbda054\") " pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:03.967899 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:03.967876 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-v7494" Oct 29 21:48:04.268816 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:04.268751 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-v7494"] Oct 29 21:48:04.272131 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:48:04.272101 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedbe3b3_df9c_49da_834e_48d64cbda054.slice/crio-8ebf5fb642f9ec90a86aaa20e0ea465c801295e68c3bf44a13e9bb8442fafa67 WatchSource:0}: Error finding container 8ebf5fb642f9ec90a86aaa20e0ea465c801295e68c3bf44a13e9bb8442fafa67: Status 404 returned error can't find the container with id 8ebf5fb642f9ec90a86aaa20e0ea465c801295e68c3bf44a13e9bb8442fafa67 Oct 29 21:48:04.691045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:04.690956 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-v7494" event={"ID":"bedbe3b3-df9c-49da-834e-48d64cbda054","Type":"ContainerStarted","Data":"185066f1c9b2efa07d6a1e9dbe626fdd1c66fc9d4306d685be4a2158b5385002"} Oct 29 21:48:04.691045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:04.691004 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-v7494" event={"ID":"bedbe3b3-df9c-49da-834e-48d64cbda054","Type":"ContainerStarted","Data":"8ebf5fb642f9ec90a86aaa20e0ea465c801295e68c3bf44a13e9bb8442fafa67"} Oct 29 21:48:06.935553 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.935499 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-v7494" podStartSLOduration=3.935484416 podStartE2EDuration="3.935484416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:04.734679532 +0000 UTC m=+346.476610206" watchObservedRunningTime="2026-10-29 21:48:06.935484416 +0000 UTC m=+348.677415089" Oct 29 21:48:06.936037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.935659 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9"] Oct 29 21:48:06.939086 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.939048 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:06.965655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.965629 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9"] Oct 29 21:48:06.976195 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.976165 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 21:48:06.976195 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.976181 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 29 21:48:06.976195 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.976194 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 21:48:06.976383 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.976169 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 29 21:48:06.976383 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.976330 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jjjl6\"" Oct 29 21:48:06.993510 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.993483 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/016e6143-39a6-45b3-ac25-cce513d56269-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:06.993614 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.993527 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fft\" (UniqueName: \"kubernetes.io/projected/016e6143-39a6-45b3-ac25-cce513d56269-kube-api-access-j9fft\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:06.993614 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:06.993584 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/016e6143-39a6-45b3-ac25-cce513d56269-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.094027 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.094000 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fft\" (UniqueName: \"kubernetes.io/projected/016e6143-39a6-45b3-ac25-cce513d56269-kube-api-access-j9fft\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.094175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.094048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/016e6143-39a6-45b3-ac25-cce513d56269-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.094175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.094150 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/016e6143-39a6-45b3-ac25-cce513d56269-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.094737 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.094716 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/016e6143-39a6-45b3-ac25-cce513d56269-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.096784 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.096762 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/016e6143-39a6-45b3-ac25-cce513d56269-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.118371 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.118343 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fft\" (UniqueName: \"kubernetes.io/projected/016e6143-39a6-45b3-ac25-cce513d56269-kube-api-access-j9fft\") pod \"kuadrant-console-plugin-66c4fb575d-s2sn9\" (UID: \"016e6143-39a6-45b3-ac25-cce513d56269\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.250589 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.250566 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" Oct 29 21:48:07.550925 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.550859 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9"] Oct 29 21:48:07.554333 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:48:07.554306 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod016e6143_39a6_45b3_ac25_cce513d56269.slice/crio-dce8b7476cff6c5ee6c6df6f6c02bd3347733d767765ff5a3c9e09d89d30c222 WatchSource:0}: Error finding container dce8b7476cff6c5ee6c6df6f6c02bd3347733d767765ff5a3c9e09d89d30c222: Status 404 returned error can't find the container with id dce8b7476cff6c5ee6c6df6f6c02bd3347733d767765ff5a3c9e09d89d30c222 Oct 29 21:48:07.666232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.666212 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-qjv5n" Oct 29 21:48:07.705268 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:07.705240 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" event={"ID":"016e6143-39a6-45b3-ac25-cce513d56269","Type":"ContainerStarted","Data":"dce8b7476cff6c5ee6c6df6f6c02bd3347733d767765ff5a3c9e09d89d30c222"} Oct 29 21:48:12.730717 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:12.730676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" event={"ID":"016e6143-39a6-45b3-ac25-cce513d56269","Type":"ContainerStarted","Data":"37279fad9054ed5773dc938454ae2e70d157f2d47cd0dee98b7222fb40e7b420"} Oct 29 21:48:12.774785 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:12.774733 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s2sn9" podStartSLOduration=2.309688391 podStartE2EDuration="6.774718754s" totalImagesPullingTime="4.465030363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:12.77352342 +0000 UTC m=+354.515454093" watchObservedRunningTime="2026-10-29 21:48:12.774718754 +0000 UTC m=+354.516649426" Oct 29 21:48:27.463463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.463433 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr"] Oct 29 21:48:27.526009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.525980 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr"] Oct 29 21:48:27.526167 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.526068 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.545335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.545312 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 21:48:27.545450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.545346 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 21:48:27.608317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.608291 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 21:48:27.608537 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.608522 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5v6tw\"" Oct 29 21:48:27.608537 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.608532 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 21:48:27.642846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.642821 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a17112fb-d40c-4e98-903f-7100f859628b-tmp\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.642996 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.642857 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl69r\" (UniqueName: \"kubernetes.io/projected/a17112fb-d40c-4e98-903f-7100f859628b-kube-api-access-fl69r\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.642996 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.642887 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17112fb-d40c-4e98-903f-7100f859628b-tls-certs\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.744045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.743991 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl69r\" (UniqueName: \"kubernetes.io/projected/a17112fb-d40c-4e98-903f-7100f859628b-kube-api-access-fl69r\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.744045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.744036 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17112fb-d40c-4e98-903f-7100f859628b-tls-certs\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.744212 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.744132 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a17112fb-d40c-4e98-903f-7100f859628b-tmp\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.746377 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.746355 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a17112fb-d40c-4e98-903f-7100f859628b-tmp\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.746644 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.746627 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17112fb-d40c-4e98-903f-7100f859628b-tls-certs\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.769020 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.768992 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl69r\" (UniqueName: \"kubernetes.io/projected/a17112fb-d40c-4e98-903f-7100f859628b-kube-api-access-fl69r\") pod \"kube-auth-proxy-698579f7b7-vq2rr\" (UID: \"a17112fb-d40c-4e98-903f-7100f859628b\") " pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.836402 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:27.836380 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" Oct 29 21:48:27.958392 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:48:27.958360 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:48:28.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:28.137200 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr"] Oct 29 21:48:28.140506 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:48:28.140476 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17112fb_d40c_4e98_903f_7100f859628b.slice/crio-300436a85646de169d574a241282226fd1c1a9a492122c4e8acb1f384b020685 WatchSource:0}: Error finding container 300436a85646de169d574a241282226fd1c1a9a492122c4e8acb1f384b020685: Status 404 returned error can't find the container with id 300436a85646de169d574a241282226fd1c1a9a492122c4e8acb1f384b020685 Oct 29 21:48:28.796494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:28.796452 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" event={"ID":"a17112fb-d40c-4e98-903f-7100f859628b","Type":"ContainerStarted","Data":"300436a85646de169d574a241282226fd1c1a9a492122c4e8acb1f384b020685"} Oct 29 21:48:30.507323 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.507285 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:48:30.510966 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.510947 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:48:30.526240 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.526217 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:48:30.536906 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.536887 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k7sdt\"" Oct 29 21:48:30.668536 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.668508 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkn8\" (UniqueName: \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\") pod \"authorino-56c6c9ddd5-4qm4t\" (UID: \"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:48:30.769706 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.769673 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkn8\" (UniqueName: \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\") pod \"authorino-56c6c9ddd5-4qm4t\" (UID: \"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:48:30.794450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.794420 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkn8\" (UniqueName: \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\") pod \"authorino-56c6c9ddd5-4qm4t\" (UID: \"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:48:30.806649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.806616 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" event={"ID":"a17112fb-d40c-4e98-903f-7100f859628b","Type":"ContainerStarted","Data":"1104246a17e4f10757429710d4c1fe4cabafdb4f430b0e03fff1904598d51ee0"} Oct 29 21:48:30.822184 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.822162 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:48:30.856421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:30.856375 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-698579f7b7-vq2rr" podStartSLOduration=1.558633703 podStartE2EDuration="3.856362349s" totalImagesPullingTime="2.297728646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:30.855134524 +0000 UTC m=+372.597065193" watchObservedRunningTime="2026-10-29 21:48:30.856362349 +0000 UTC m=+372.598293021" Oct 29 21:48:31.125256 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:31.125191 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:48:31.128282 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:48:31.128251 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea005b3_6c49_4168_8d1f_a0a5d26e89ad.slice/crio-d69e8af52b6ce2f76747abf0f2b9524caceb56bc1ffb0d046ade10ab8e0ec53e WatchSource:0}: Error finding container d69e8af52b6ce2f76747abf0f2b9524caceb56bc1ffb0d046ade10ab8e0ec53e: Status 404 returned error can't find the container with id d69e8af52b6ce2f76747abf0f2b9524caceb56bc1ffb0d046ade10ab8e0ec53e Oct 29 21:48:31.813400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:31.813332 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" event={"ID":"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad","Type":"ContainerStarted","Data":"d69e8af52b6ce2f76747abf0f2b9524caceb56bc1ffb0d046ade10ab8e0ec53e"} Oct 29 21:48:33.101119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.099705 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc"] Oct 29 21:48:33.107010 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.106986 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.122902 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.122879 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc"] Oct 29 21:48:33.133485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.133462 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 29 21:48:33.134695 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.134674 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 29 21:48:33.134800 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.134692 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 29 21:48:33.134800 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.134773 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-jr7v7\"" Oct 29 21:48:33.190216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.190190 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-metrics-certs\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.190343 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.190219 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-cert\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.190409 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.190362 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd958\" (UniqueName: \"kubernetes.io/projected/feb24be2-6871-48e7-adb6-ff2b49d8e813-kube-api-access-cd958\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.190453 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.190410 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feb24be2-6871-48e7-adb6-ff2b49d8e813-manager-config\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.290940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.290905 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-metrics-certs\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.291125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.290944 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-cert\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.291125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.291025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd958\" (UniqueName: \"kubernetes.io/projected/feb24be2-6871-48e7-adb6-ff2b49d8e813-kube-api-access-cd958\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.291125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.291060 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feb24be2-6871-48e7-adb6-ff2b49d8e813-manager-config\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.291957 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.291922 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feb24be2-6871-48e7-adb6-ff2b49d8e813-manager-config\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.294023 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.294003 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-metrics-certs\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.294309 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.294287 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feb24be2-6871-48e7-adb6-ff2b49d8e813-cert\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.316687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.316666 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd958\" (UniqueName: \"kubernetes.io/projected/feb24be2-6871-48e7-adb6-ff2b49d8e813-kube-api-access-cd958\") pod \"jobset-controller-manager-784fb6ccb9-rf7nc\" (UID: \"feb24be2-6871-48e7-adb6-ff2b49d8e813\") " pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.422850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.422769 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:33.753511 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.753485 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc"] Oct 29 21:48:33.757090 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:48:33.757047 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb24be2_6871_48e7_adb6_ff2b49d8e813.slice/crio-557aea30feb8757406342a02b2c217c8b443afbae42c2e42588bee8e067063e6 WatchSource:0}: Error finding container 557aea30feb8757406342a02b2c217c8b443afbae42c2e42588bee8e067063e6: Status 404 returned error can't find the container with id 557aea30feb8757406342a02b2c217c8b443afbae42c2e42588bee8e067063e6 Oct 29 21:48:33.823198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.823165 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" event={"ID":"feb24be2-6871-48e7-adb6-ff2b49d8e813","Type":"ContainerStarted","Data":"557aea30feb8757406342a02b2c217c8b443afbae42c2e42588bee8e067063e6"} Oct 29 21:48:33.825123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.825099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" event={"ID":"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad","Type":"ContainerStarted","Data":"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf"} Oct 29 21:48:33.864504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:33.864460 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" podStartSLOduration=1.261298913 podStartE2EDuration="3.864448845s" totalImagesPullingTime="2.603149932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:33.86440474 +0000 UTC m=+375.606335414" watchObservedRunningTime="2026-10-29 21:48:33.864448845 +0000 UTC m=+375.606379515" Oct 29 21:48:36.845455 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:36.845412 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" event={"ID":"feb24be2-6871-48e7-adb6-ff2b49d8e813","Type":"ContainerStarted","Data":"bd620df7efaf2eb95f208b8739b0c949bd2e3714c528a7e4cf5ab8cbd90d834a"} Oct 29 21:48:36.845939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:36.845696 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:36.891486 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:36.891436 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" podStartSLOduration=1.7328041330000001 podStartE2EDuration="3.891419821s" totalImagesPullingTime="2.158615688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:36.890327077 +0000 UTC m=+378.632257728" watchObservedRunningTime="2026-10-29 21:48:36.891419821 +0000 UTC m=+378.633350497" Oct 29 21:48:47.854815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:48:47.854784 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-784fb6ccb9-rf7nc" Oct 29 21:48:57.963034 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:48:57.962972 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:49:27.962751 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:49:27.962711 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:49:56.408608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.408568 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:49:56.411568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.411551 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.433002 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.432976 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:49:56.444235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.444215 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 29 21:49:56.506224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.506201 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz8c\" (UniqueName: \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.506351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.506329 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.607312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.607282 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.607452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.607330 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz8c\" (UniqueName: \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.609858 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.609840 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.632388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.632357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz8c\" (UniqueName: \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\") pod \"authorino-775bdf9c76-bmxwg\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:56.721408 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:56.721383 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:49:57.023161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:57.023057 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:49:57.026114 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:49:57.026083 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d041fb_1394_459a_b9bf_b24ad7266e27.slice/crio-dc2fd62862d2b20cd59752dfbca113d7a82c42e490522f59997d228b6ddc9d3b WatchSource:0}: Error finding container dc2fd62862d2b20cd59752dfbca113d7a82c42e490522f59997d228b6ddc9d3b: Status 404 returned error can't find the container with id dc2fd62862d2b20cd59752dfbca113d7a82c42e490522f59997d228b6ddc9d3b Oct 29 21:49:57.190211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:57.190176 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" event={"ID":"a4d041fb-1394-459a-b9bf-b24ad7266e27","Type":"ContainerStarted","Data":"dc2fd62862d2b20cd59752dfbca113d7a82c42e490522f59997d228b6ddc9d3b"} Oct 29 21:49:57.965602 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:49:57.965557 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:49:58.195943 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.195909 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" event={"ID":"a4d041fb-1394-459a-b9bf-b24ad7266e27","Type":"ContainerStarted","Data":"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1"} Oct 29 21:49:58.237318 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.237239 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" podStartSLOduration=1.929726084 podStartE2EDuration="2.237224721s" totalImagesPullingTime="307.498637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:58.23603082 +0000 UTC m=+459.977961493" watchObservedRunningTime="2026-10-29 21:49:58.237224721 +0000 UTC m=+459.979155393" Oct 29 21:49:58.266442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.266413 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:49:58.266601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.266582 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" podUID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" containerName="authorino" containerID="cri-o://d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf" gracePeriod=30 Oct 29 21:49:58.512503 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.512481 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:49:58.623828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.623801 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\" (UniqueName: \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\") pod \"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad\" (UID: \"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad\") " Oct 29 21:49:58.626036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.626015 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8" pod "1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" (UID: "1ea005b3-6c49-4168-8d1f-a0a5d26e89ad"). InnerVolumeSpecName "kube-api-access-hnkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:58.724603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:58.724573 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkn8\" (UniqueName: \"kubernetes.io/projected/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad-kube-api-access-hnkn8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:49:59.201219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.201185 4543 generic.go:356] "Generic (PLEG): container finished" podID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" containerID="d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf" exitCode=0 Oct 29 21:49:59.201604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.201237 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" Oct 29 21:49:59.201604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.201274 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" event={"ID":"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad","Type":"ContainerDied","Data":"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf"} Oct 29 21:49:59.201604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.201317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qm4t" event={"ID":"1ea005b3-6c49-4168-8d1f-a0a5d26e89ad","Type":"ContainerDied","Data":"d69e8af52b6ce2f76747abf0f2b9524caceb56bc1ffb0d046ade10ab8e0ec53e"} Oct 29 21:49:59.201604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.201337 4543 scope.go:122] "RemoveContainer" containerID="d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf" Oct 29 21:49:59.210835 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.210818 4543 scope.go:122] "RemoveContainer" containerID="d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf" Oct 29 21:49:59.211146 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:49:59.211122 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf\": container with ID starting with d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf not found: ID does not exist" containerID="d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf" Oct 29 21:49:59.211208 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.211159 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf"} err="failed to get container status \"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf\": rpc error: code = NotFound desc = could not find container \"d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf\": container with ID starting with d1f7cbe579321f697bd60c77292b5c768f9ba8ccffe1114f0001a406331dc9bf not found: ID does not exist" Oct 29 21:49:59.271999 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.271979 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:49:59.284238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:49:59.284220 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qm4t"] Oct 29 21:50:00.383561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:50:00.383524 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" path="/var/lib/kubelet/pods/1ea005b3-6c49-4168-8d1f-a0a5d26e89ad/volumes" Oct 29 21:50:27.962805 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:50:27.962754 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:50:57.963039 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:50:57.962946 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:51:27.961055 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:51:27.961016 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:51:57.962230 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:51:57.962186 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:52:02.502263 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.502174 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9"] Oct 29 21:52:02.505025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.502497 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" containerName="authorino" Oct 29 21:52:02.505025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.502508 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" containerName="authorino" Oct 29 21:52:02.505025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.502596 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea005b3-6c49-4168-8d1f-a0a5d26e89ad" containerName="authorino" Oct 29 21:52:02.505583 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.505564 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524462 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524592 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e19b48-3028-4018-88ec-ce30df35e7bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524592 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524515 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524592 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524572 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzkc\" (UniqueName: \"kubernetes.io/projected/83e19b48-3028-4018-88ec-ce30df35e7bd-kube-api-access-bvzkc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524717 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524695 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.524789 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.524773 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.536307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.536278 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwg6\"" Oct 29 21:52:02.536427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.536318 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 21:52:02.536427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.536391 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 29 21:52:02.536562 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.536459 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 21:52:02.538511 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.538492 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9"] Oct 29 21:52:02.625312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625270 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzkc\" (UniqueName: \"kubernetes.io/projected/83e19b48-3028-4018-88ec-ce30df35e7bd-kube-api-access-bvzkc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625341 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625406 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625451 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e19b48-3028-4018-88ec-ce30df35e7bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625713 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625510 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.625864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.625840 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.626272 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.626235 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.626394 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.626357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.627816 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.627793 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e19b48-3028-4018-88ec-ce30df35e7bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.628181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.628160 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e19b48-3028-4018-88ec-ce30df35e7bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.656936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.656912 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzkc\" (UniqueName: \"kubernetes.io/projected/83e19b48-3028-4018-88ec-ce30df35e7bd-kube-api-access-bvzkc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xbmp9\" (UID: \"83e19b48-3028-4018-88ec-ce30df35e7bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:02.818742 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:02.818663 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:03.125955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.125928 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9"] Oct 29 21:52:03.128682 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:03.128653 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e19b48_3028_4018_88ec_ce30df35e7bd.slice/crio-93a7adaf39cb43e7c24e7ea6dfaa52b74f626edbdca45fe855f1c8fb41ae41d4 WatchSource:0}: Error finding container 93a7adaf39cb43e7c24e7ea6dfaa52b74f626edbdca45fe855f1c8fb41ae41d4: Status 404 returned error can't find the container with id 93a7adaf39cb43e7c24e7ea6dfaa52b74f626edbdca45fe855f1c8fb41ae41d4 Oct 29 21:52:03.742219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.742152 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" event={"ID":"83e19b48-3028-4018-88ec-ce30df35e7bd","Type":"ContainerStarted","Data":"93a7adaf39cb43e7c24e7ea6dfaa52b74f626edbdca45fe855f1c8fb41ae41d4"} Oct 29 21:52:03.746634 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.746603 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2"] Oct 29 21:52:03.752188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.752161 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.771357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.770960 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2"] Oct 29 21:52:03.780914 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.780892 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 29 21:52:03.932873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.932840 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.933052 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.932886 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.933052 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.932994 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f8925d-8433-499f-9523-93d536c176f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.933209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.933113 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.933209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.933151 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:03.933317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:03.933272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq46p\" (UniqueName: \"kubernetes.io/projected/40f8925d-8433-499f-9523-93d536c176f2-kube-api-access-gq46p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034017 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.033942 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f8925d-8433-499f-9523-93d536c176f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034017 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034005 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034034 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034111 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq46p\" (UniqueName: \"kubernetes.io/projected/40f8925d-8433-499f-9523-93d536c176f2-kube-api-access-gq46p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034138 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034167 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034406 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034712 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.034877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.034852 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.036767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.036745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f8925d-8433-499f-9523-93d536c176f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.036856 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.036743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f8925d-8433-499f-9523-93d536c176f2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.059127 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.059102 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq46p\" (UniqueName: \"kubernetes.io/projected/40f8925d-8433-499f-9523-93d536c176f2-kube-api-access-gq46p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f4xp2\" (UID: \"40f8925d-8433-499f-9523-93d536c176f2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.072766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.072745 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:04.383909 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:04.383884 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f8925d_8433_499f_9523_93d536c176f2.slice/crio-7ca2c0a6714f2417d2fb84dee92dc057f35f5dfd5dc7b7bbd5fa1190977ba752 WatchSource:0}: Error finding container 7ca2c0a6714f2417d2fb84dee92dc057f35f5dfd5dc7b7bbd5fa1190977ba752: Status 404 returned error can't find the container with id 7ca2c0a6714f2417d2fb84dee92dc057f35f5dfd5dc7b7bbd5fa1190977ba752 Oct 29 21:52:04.385206 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.385183 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2"] Oct 29 21:52:04.749967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.749930 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" event={"ID":"83e19b48-3028-4018-88ec-ce30df35e7bd","Type":"ContainerStarted","Data":"0caf8ee782998fbdbd073b4b1fa6919b82c20cae91b87a64b15dc1412a1dd05f"} Oct 29 21:52:04.751616 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.751588 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" event={"ID":"40f8925d-8433-499f-9523-93d536c176f2","Type":"ContainerStarted","Data":"0a72e316e14580b24c19806cc75ed7b481c3cb996ea9b08b5eb395b76ae8b253"} Oct 29 21:52:04.751739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.751629 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" event={"ID":"40f8925d-8433-499f-9523-93d536c176f2","Type":"ContainerStarted","Data":"7ca2c0a6714f2417d2fb84dee92dc057f35f5dfd5dc7b7bbd5fa1190977ba752"} Oct 29 21:52:04.793601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.793538 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" podStartSLOduration=1.605591786 podStartE2EDuration="2.793518505s" totalImagesPullingTime="1.187926719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:04.792039525 +0000 UTC m=+586.533970199" watchObservedRunningTime="2026-10-29 21:52:04.793518505 +0000 UTC m=+586.535449179" Oct 29 21:52:04.819720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.819690 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:04.835323 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:04.835280 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" podStartSLOduration=1.6272729259999998 podStartE2EDuration="1.835268446s" totalImagesPullingTime="207.99552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:04.833206801 +0000 UTC m=+586.575137474" watchObservedRunningTime="2026-10-29 21:52:04.835268446 +0000 UTC m=+586.577199119" Oct 29 21:52:05.073647 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:05.073609 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:12.819263 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:12.819228 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:12.832144 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:12.832119 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:13.674859 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.674825 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r"] Oct 29 21:52:13.678359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.678337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.696152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.696128 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 29 21:52:13.708652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.708628 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r"] Oct 29 21:52:13.722292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722260 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.722433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722318 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.722433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722375 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.722433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722394 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.722433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722411 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggqg\" (UniqueName: \"kubernetes.io/projected/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-kube-api-access-rggqg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.722692 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.722439 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.803660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.803633 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xbmp9" Oct 29 21:52:13.822795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822768 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822829 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822861 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822885 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822912 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggqg\" (UniqueName: \"kubernetes.io/projected/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-kube-api-access-rggqg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.822974 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.823303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.823357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.823502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.823486 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.825364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.825345 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.825763 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.825742 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.847257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.847232 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggqg\" (UniqueName: \"kubernetes.io/projected/e92f01a9-e0b3-4c27-9ed2-63e20b27df7a-kube-api-access-rggqg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r\" (UID: \"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:13.997758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:13.997716 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:14.073620 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.073575 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:14.090843 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.090821 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:14.300630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.300547 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r"] Oct 29 21:52:14.304826 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:14.304792 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92f01a9_e0b3_4c27_9ed2_63e20b27df7a.slice/crio-36f63cc53115dcc192d424a6c79aedc48abccfe1f968fc272372e6f68cf29e3b WatchSource:0}: Error finding container 36f63cc53115dcc192d424a6c79aedc48abccfe1f968fc272372e6f68cf29e3b: Status 404 returned error can't find the container with id 36f63cc53115dcc192d424a6c79aedc48abccfe1f968fc272372e6f68cf29e3b Oct 29 21:52:14.797951 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.797910 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" event={"ID":"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a","Type":"ContainerStarted","Data":"3f8c09b455a313bf98363b3a0b8abfb7af101f8f6877bfe61d786c6380bb3f40"} Oct 29 21:52:14.798159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.797998 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" event={"ID":"e92f01a9-e0b3-4c27-9ed2-63e20b27df7a","Type":"ContainerStarted","Data":"36f63cc53115dcc192d424a6c79aedc48abccfe1f968fc272372e6f68cf29e3b"} Oct 29 21:52:14.811640 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.811615 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f4xp2" Oct 29 21:52:14.866782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.866707 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" podStartSLOduration=1.6374649049999999 podStartE2EDuration="1.866694478s" totalImagesPullingTime="229.229573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:14.865145267 +0000 UTC m=+596.607075940" watchObservedRunningTime="2026-10-29 21:52:14.866694478 +0000 UTC m=+596.608625151" Oct 29 21:52:14.998535 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:14.998510 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:15.394028 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.393995 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l"] Oct 29 21:52:15.397408 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.397387 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.418621 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.418595 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l"] Oct 29 21:52:15.426297 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.426275 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 29 21:52:15.433877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.433849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.434006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.433983 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.434061 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.434020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.434183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.434156 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfv22\" (UniqueName: \"kubernetes.io/projected/e386a9e9-223d-4c2d-96c2-148d940e4044-kube-api-access-gfv22\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.434289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.434266 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.434335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.434316 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e386a9e9-223d-4c2d-96c2-148d940e4044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535259 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535236 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfv22\" (UniqueName: \"kubernetes.io/projected/e386a9e9-223d-4c2d-96c2-148d940e4044-kube-api-access-gfv22\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535276 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535298 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e386a9e9-223d-4c2d-96c2-148d940e4044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535315 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535362 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535378 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535787 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.535903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.535870 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.537652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.537632 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e386a9e9-223d-4c2d-96c2-148d940e4044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.538013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.537993 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e386a9e9-223d-4c2d-96c2-148d940e4044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.559474 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.559449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfv22\" (UniqueName: \"kubernetes.io/projected/e386a9e9-223d-4c2d-96c2-148d940e4044-kube-api-access-gfv22\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l\" (UID: \"e386a9e9-223d-4c2d-96c2-148d940e4044\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:15.711096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:15.711044 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:16.015627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:16.015539 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l"] Oct 29 21:52:16.019692 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:16.019663 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode386a9e9_223d_4c2d_96c2_148d940e4044.slice/crio-ef6b5faaeb818d5712ad33e1b1ff5d9f63fe0460047f89ee0086d60ed22c2825 WatchSource:0}: Error finding container ef6b5faaeb818d5712ad33e1b1ff5d9f63fe0460047f89ee0086d60ed22c2825: Status 404 returned error can't find the container with id ef6b5faaeb818d5712ad33e1b1ff5d9f63fe0460047f89ee0086d60ed22c2825 Oct 29 21:52:16.813280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:16.813243 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" event={"ID":"e386a9e9-223d-4c2d-96c2-148d940e4044","Type":"ContainerStarted","Data":"de134df3dee15c137a4c96bbf23e2951967367c9b9f8c6a7c1d8144c28d84278"} Oct 29 21:52:16.813477 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:16.813292 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" event={"ID":"e386a9e9-223d-4c2d-96c2-148d940e4044","Type":"ContainerStarted","Data":"ef6b5faaeb818d5712ad33e1b1ff5d9f63fe0460047f89ee0086d60ed22c2825"} Oct 29 21:52:16.856098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:16.856031 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podStartSLOduration=1.63713922 podStartE2EDuration="1.85600739s" totalImagesPullingTime="218.86817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:16.855761764 +0000 UTC m=+598.597692438" watchObservedRunningTime="2026-10-29 21:52:16.85600739 +0000 UTC m=+598.597938064" Oct 29 21:52:17.711564 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:17.711534 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:21.869767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.869730 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 21:52:21.873738 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.873714 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888518 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888557 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888620 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888757 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888649 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888777 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888861 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888823 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888907 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888886 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.888944 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888926 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.889009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.888949 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7xm\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.889673 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.889656 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 21:52:21.889673 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.889664 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4b7xt\"" Oct 29 21:52:21.902606 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.902586 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 21:52:21.989404 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989379 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989415 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989440 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7xm\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989502 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989534 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989605 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.989973 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.989946 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.990103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.990058 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.990356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.990334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.990418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.990351 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.990418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.990351 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.991735 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.991715 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:21.992189 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:21.992172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:22.013543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:22.013517 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:22.013623 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:22.013603 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7xm\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\") pod \"maas-default-gateway-openshift-default-6f54779779-fmz5k\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:22.186185 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:22.186133 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:22.500875 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:22.500848 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 21:52:22.504145 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:22.504118 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c78f57_8e7a_438e_997a_a3df95e5a7ff.slice/crio-9bd8361f929f99696a9605506dbb60782f65dba2884357db377d37fbd0d6d6cc WatchSource:0}: Error finding container 9bd8361f929f99696a9605506dbb60782f65dba2884357db377d37fbd0d6d6cc: Status 404 returned error can't find the container with id 9bd8361f929f99696a9605506dbb60782f65dba2884357db377d37fbd0d6d6cc Oct 29 21:52:22.840195 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:22.840128 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" event={"ID":"e3c78f57-8e7a-438e-997a-a3df95e5a7ff","Type":"ContainerStarted","Data":"9bd8361f929f99696a9605506dbb60782f65dba2884357db377d37fbd0d6d6cc"} Oct 29 21:52:23.998244 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:23.998215 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:24.017857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:24.017830 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:24.862113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:24.862061 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r" Oct 29 21:52:24.984743 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:24.984710 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:24.984814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:24.984776 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:24.984814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:24.984804 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:25.712138 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:25.712099 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:25.724542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:25.724514 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:25.859967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:25.859936 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" event={"ID":"e3c78f57-8e7a-438e-997a-a3df95e5a7ff","Type":"ContainerStarted","Data":"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3"} Oct 29 21:52:25.871422 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:25.871398 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:52:25.906715 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:25.906670 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podStartSLOduration=2.428872089 podStartE2EDuration="4.9066576s" totalImagesPullingTime="2.477785511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:25.905364519 +0000 UTC m=+607.647295191" watchObservedRunningTime="2026-10-29 21:52:25.9066576 +0000 UTC m=+607.648588273" Oct 29 21:52:26.186888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:26.186789 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:26.188524 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:26.188488 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:26.188680 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:26.188543 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:27.186699 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:27.186664 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:27.187200 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:27.186723 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:27.979660 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:52:27.979626 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:52:28.186551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:28.186519 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:28.186702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:28.186571 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:29.186948 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:29.186915 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:29.187438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:29.186969 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:30.186716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:30.186676 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:30.186888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:30.186739 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:31.186841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:31.186808 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:31.187225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:31.186866 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:32.187049 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:32.187013 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 21:52:32.187439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:32.187113 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 21:52:33.189945 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:33.189919 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:33.190382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:33.190175 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:33.190712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:33.190697 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 21:52:36.945481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:36.945444 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 21:52:36.949681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:36.949658 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:36.980303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:36.980274 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 21:52:37.046421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046387 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046459 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046510 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046598 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046642 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4phc\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046674 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046858 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046779 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046858 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.046932 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.046868 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.050719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.050693 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:52:37.050896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.050872 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" podUID="a4d041fb-1394-459a-b9bf-b24ad7266e27" containerName="authorino" containerID="cri-o://afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1" gracePeriod=30 Oct 29 21:52:37.147530 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147500 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147535 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147583 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147630 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147692 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147746 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147785 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147965 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147815 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4phc\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147965 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.147965 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.147906 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.148289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.148253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.148289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.148272 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.148428 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.148291 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.148466 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.148426 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.149988 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.149964 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.150368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.150346 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.174892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.174872 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4phc\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.177400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.177381 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nmzq9\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.262582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.262555 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:37.286620 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.286598 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:52:37.349623 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.349599 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\" (UniqueName: \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\") pod \"a4d041fb-1394-459a-b9bf-b24ad7266e27\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " Oct 29 21:52:37.349751 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.349688 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\" (UniqueName: \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\") pod \"a4d041fb-1394-459a-b9bf-b24ad7266e27\" (UID: \"a4d041fb-1394-459a-b9bf-b24ad7266e27\") " Oct 29 21:52:37.351688 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.351665 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c" pod "a4d041fb-1394-459a-b9bf-b24ad7266e27" (UID: "a4d041fb-1394-459a-b9bf-b24ad7266e27"). InnerVolumeSpecName "kube-api-access-dkz8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:52:37.359910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.359885 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert" pod "a4d041fb-1394-459a-b9bf-b24ad7266e27" (UID: "a4d041fb-1394-459a-b9bf-b24ad7266e27"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:52:37.450641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.450613 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz8c\" (UniqueName: \"kubernetes.io/projected/a4d041fb-1394-459a-b9bf-b24ad7266e27-kube-api-access-dkz8c\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:52:37.450641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.450641 4543 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4d041fb-1394-459a-b9bf-b24ad7266e27-tls-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:52:37.579455 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.579385 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 21:52:37.582427 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:37.582405 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4783af05_3ee3_409a_a387_8c4a692c3d7e.slice/crio-3887123cbfe3933120c790f42066cb96dbce64aa4dbcd5d050cc14096fe09872 WatchSource:0}: Error finding container 3887123cbfe3933120c790f42066cb96dbce64aa4dbcd5d050cc14096fe09872: Status 404 returned error can't find the container with id 3887123cbfe3933120c790f42066cb96dbce64aa4dbcd5d050cc14096fe09872 Oct 29 21:52:37.585471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.585442 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:37.585550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.585504 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:37.585550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.585531 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:37.912591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.912509 4543 generic.go:356] "Generic (PLEG): container finished" podID="a4d041fb-1394-459a-b9bf-b24ad7266e27" containerID="afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1" exitCode=0 Oct 29 21:52:37.912591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.912554 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" Oct 29 21:52:37.912776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.912592 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" event={"ID":"a4d041fb-1394-459a-b9bf-b24ad7266e27","Type":"ContainerDied","Data":"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1"} Oct 29 21:52:37.912776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.912639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775bdf9c76-bmxwg" event={"ID":"a4d041fb-1394-459a-b9bf-b24ad7266e27","Type":"ContainerDied","Data":"dc2fd62862d2b20cd59752dfbca113d7a82c42e490522f59997d228b6ddc9d3b"} Oct 29 21:52:37.912776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.912658 4543 scope.go:122] "RemoveContainer" containerID="afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1" Oct 29 21:52:37.914271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.914245 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" event={"ID":"4783af05-3ee3-409a-a387-8c4a692c3d7e","Type":"ContainerStarted","Data":"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b"} Oct 29 21:52:37.914397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.914294 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" event={"ID":"4783af05-3ee3-409a-a387-8c4a692c3d7e","Type":"ContainerStarted","Data":"3887123cbfe3933120c790f42066cb96dbce64aa4dbcd5d050cc14096fe09872"} Oct 29 21:52:37.921876 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.921859 4543 scope.go:122] "RemoveContainer" containerID="afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1" Oct 29 21:52:37.922123 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:52:37.922105 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1\": container with ID starting with afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1 not found: ID does not exist" containerID="afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1" Oct 29 21:52:37.922179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.922131 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1"} err="failed to get container status \"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1\": rpc error: code = NotFound desc = could not find container \"afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1\": container with ID starting with afcf0e4ce1d7b0a096dd5f51b9b57f5d1f474169ce3f05fcdae44549b720ffc1 not found: ID does not exist" Oct 29 21:52:37.964531 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.964510 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:52:37.976626 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:37.976607 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-775bdf9c76-bmxwg"] Oct 29 21:52:38.018928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:38.018878 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podStartSLOduration=2.018861231 podStartE2EDuration="2.018861231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:38.017207889 +0000 UTC m=+619.759138563" watchObservedRunningTime="2026-10-29 21:52:38.018861231 +0000 UTC m=+619.760791906" Oct 29 21:52:38.262900 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:38.262857 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:38.264494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:38.264466 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:38.264612 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:38.264523 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:38.382826 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:38.382797 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d041fb-1394-459a-b9bf-b24ad7266e27" path="/var/lib/kubelet/pods/a4d041fb-1394-459a-b9bf-b24ad7266e27/volumes" Oct 29 21:52:39.263526 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:39.263491 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:39.264016 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:39.263550 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:40.263304 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:40.263266 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:40.263469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:40.263319 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:41.263194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:41.263158 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:41.263615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:41.263209 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:42.263379 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:42.263344 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:42.263723 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:42.263394 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:43.263511 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:43.263478 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:43.263894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:43.263528 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:44.263005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:44.262972 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 21:52:44.263216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:44.263024 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 21:52:45.845504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:45.845470 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:45.845980 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:45.845753 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:45.846575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:45.846553 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 21:52:51.983884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:51.983846 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 21:52:51.984316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:51.984225 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a4d041fb-1394-459a-b9bf-b24ad7266e27" containerName="authorino" Oct 29 21:52:51.984316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:51.984240 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d041fb-1394-459a-b9bf-b24ad7266e27" containerName="authorino" Oct 29 21:52:51.984400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:51.984325 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d041fb-1394-459a-b9bf-b24ad7266e27" containerName="authorino" Oct 29 21:52:51.986389 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:51.986367 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.017562 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.017536 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 21:52:52.071570 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071544 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqqd\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071696 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071605 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071764 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071764 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071739 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071835 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071855 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.071911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071883 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.072015 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.071966 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.172821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172797 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.172939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172827 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.172939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172845 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.172939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172863 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.172939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172898 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.172952 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173015 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173127 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173185 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqqd\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173268 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173332 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173514 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173487 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173611 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.173984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.173958 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.175419 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.175395 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.175490 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.175400 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.197147 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.197126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.197276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.197260 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqqd\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\") pod \"maas-default-gateway-openshift-default-6f54779779-dnk78\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.300099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.300026 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:52.617221 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.617193 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 21:52:52.619825 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:52:52.619795 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode096a529_de3b_465f_8805_c0a68eaf0452.slice/crio-0d8f5d0ab2d9c05b831dfaed690bd2be7983890d4fa65e77ea7d2b761fa248ac WatchSource:0}: Error finding container 0d8f5d0ab2d9c05b831dfaed690bd2be7983890d4fa65e77ea7d2b761fa248ac: Status 404 returned error can't find the container with id 0d8f5d0ab2d9c05b831dfaed690bd2be7983890d4fa65e77ea7d2b761fa248ac Oct 29 21:52:52.621926 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.621882 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:52.621982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.621956 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:52.622012 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:52.621986 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:52:53.004550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.004514 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" event={"ID":"e096a529-de3b-465f-8805-c0a68eaf0452","Type":"ContainerStarted","Data":"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607"} Oct 29 21:52:53.004946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.004565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" event={"ID":"e096a529-de3b-465f-8805-c0a68eaf0452","Type":"ContainerStarted","Data":"0d8f5d0ab2d9c05b831dfaed690bd2be7983890d4fa65e77ea7d2b761fa248ac"} Oct 29 21:52:53.055002 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.054921 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podStartSLOduration=2.054900507 podStartE2EDuration="2.054900507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:53.053956252 +0000 UTC m=+634.795886925" watchObservedRunningTime="2026-10-29 21:52:53.054900507 +0000 UTC m=+634.796831183" Oct 29 21:52:53.301005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.300908 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:52:53.302596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.302563 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:53.302731 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:53.302618 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:54.301126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:54.301067 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:54.301495 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:54.301139 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:55.300688 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:55.300655 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:55.300846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:55.300705 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:56.300585 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:56.300550 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:56.300980 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:56.300604 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:57.301309 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:57.301272 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:57.301682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:57.301334 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:57.977039 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:52:57.976999 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:52:58.300850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:58.300771 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:58.300850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:58.300834 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:52:59.300585 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:59.300547 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 21:52:59.301004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:52:59.300608 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 21:53:00.724383 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:00.724351 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:53:00.724946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:00.724851 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:53:00.725818 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:00.725797 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 21:53:05.664541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.664503 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:53:05.668301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.668280 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.686496 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.686475 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5d56935a-openshift-default-dockercfg-9z8tq\"" Oct 29 21:53:05.700413 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.700391 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:53:05.811921 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.811892 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.811921 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.811927 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.811921 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.811962 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812102 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812130 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812145 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbrs\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812206 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.812469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.812452 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913675 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913647 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913679 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913696 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbrs\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913714 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913732 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913787 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.913811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913803 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913834 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.913880 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.914122 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914309 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.914172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.914342 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.914366 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.914727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.914591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.916210 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.916186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.916210 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.916196 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.938402 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.938372 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.938516 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.938409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbrs\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\") pod \"e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:05.983159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:05.983131 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:06.126216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.126181 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:06.130492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.130473 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.151485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.151462 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-229ab2ac-openshift-default-dockercfg-bzjhn\"" Oct 29 21:53:06.153190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.153165 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:06.217228 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217207 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217233 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217264 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217364 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217430 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217479 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217454 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78tt\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217548 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.217614 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.217594 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.300932 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.300891 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:53:06.305850 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:06.305815 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81b469b_69ce_4be7_8093_fec68c3365af.slice/crio-38ec3265ddb1afebac0c3c37937c3e6373ce2a7cda254b4418777165d1ec4a04 WatchSource:0}: Error finding container 38ec3265ddb1afebac0c3c37937c3e6373ce2a7cda254b4418777165d1ec4a04: Status 404 returned error can't find the container with id 38ec3265ddb1afebac0c3c37937c3e6373ce2a7cda254b4418777165d1ec4a04 Oct 29 21:53:06.308302 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.308268 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:06.308391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.308357 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:06.308441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.308392 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:06.318197 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318176 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318276 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318352 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318311 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318452 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318547 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318500 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78tt\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318564 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318593 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318624 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318998 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.318998 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.318934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.319334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.319171 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.320813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.320794 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.321248 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.321224 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.343705 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.343687 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.343774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.343753 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78tt\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\") pod \"e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.389808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.389779 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:53:06.393358 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.393336 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.439258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.429463 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f2459168-gw-openshift-default-dockercfg-n6j5z\"" Oct 29 21:53:06.439416 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.439323 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:53:06.445125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.445097 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:06.530864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.530835 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531001 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.530977 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531031 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531097 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531124 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531456 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531535 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531462 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954zl\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.531602 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.531550 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.632992 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633205 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633043 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633205 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633069 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954zl\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633205 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633126 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633205 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633167 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633226 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633328 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633354 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633656 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633598 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633656 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.633957 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.633935 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.634171 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.634047 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.634171 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.634016 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.636402 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.636374 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.636659 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.636643 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.660042 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.660012 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954zl\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.660268 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.660246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\") pod \"e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.710533 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.710510 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:06.755806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.755647 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:06.760864 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:06.760191 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46baa5cc_9e6b_488d_b81d_cbf90eb0ae0d.slice/crio-714c2685d723cfaf6695fb5cbfc180a9e6b676709f00dbcaabf79144a4cf425a WatchSource:0}: Error finding container 714c2685d723cfaf6695fb5cbfc180a9e6b676709f00dbcaabf79144a4cf425a: Status 404 returned error can't find the container with id 714c2685d723cfaf6695fb5cbfc180a9e6b676709f00dbcaabf79144a4cf425a Oct 29 21:53:06.763088 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.762404 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:06.763088 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.762491 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:06.763088 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:06.762530 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:07.015615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.015477 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:53:07.018201 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:07.018173 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f2000e_6dbe_4dda_aade_c55688d2e7ca.slice/crio-b8c13c0bfe056ea0662f5998afa1949f5ee2a3ec25cb26ebfa483f2b2b4c5607 WatchSource:0}: Error finding container b8c13c0bfe056ea0662f5998afa1949f5ee2a3ec25cb26ebfa483f2b2b4c5607: Status 404 returned error can't find the container with id b8c13c0bfe056ea0662f5998afa1949f5ee2a3ec25cb26ebfa483f2b2b4c5607 Oct 29 21:53:07.021009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.020974 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:07.021115 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.021054 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:07.021115 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.021100 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:07.075714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.075678 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerStarted","Data":"d30f2f1ba5c7f7f60fa0fbc8308736543830f2cca1548021ab10d0e0ed745426"} Oct 29 21:53:07.075899 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.075737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerStarted","Data":"714c2685d723cfaf6695fb5cbfc180a9e6b676709f00dbcaabf79144a4cf425a"} Oct 29 21:53:07.078145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.078118 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerStarted","Data":"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3"} Oct 29 21:53:07.078260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.078173 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerStarted","Data":"38ec3265ddb1afebac0c3c37937c3e6373ce2a7cda254b4418777165d1ec4a04"} Oct 29 21:53:07.079542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.079518 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerStarted","Data":"b8c13c0bfe056ea0662f5998afa1949f5ee2a3ec25cb26ebfa483f2b2b4c5607"} Oct 29 21:53:07.127363 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.127305 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podStartSLOduration=1.127289558 podStartE2EDuration="1.127289558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:07.126502632 +0000 UTC m=+648.868433317" watchObservedRunningTime="2026-10-29 21:53:07.127289558 +0000 UTC m=+648.869220236" Oct 29 21:53:07.446206 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.446118 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:07.450725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.450702 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:07.498752 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.498708 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podStartSLOduration=2.498692708 podStartE2EDuration="2.498692708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:07.179612169 +0000 UTC m=+648.921542841" watchObservedRunningTime="2026-10-29 21:53:07.498692708 +0000 UTC m=+649.240623398" Oct 29 21:53:07.983488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.983461 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:07.988559 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:07.988534 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:08.085774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.085734 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerStarted","Data":"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd"} Oct 29 21:53:08.086325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.086297 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:08.086481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.086344 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:08.087009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.086984 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:08.087137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.087044 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:08.131678 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.131633 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podStartSLOduration=2.131621484 podStartE2EDuration="2.131621484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:08.130546751 +0000 UTC m=+649.872477428" watchObservedRunningTime="2026-10-29 21:53:08.131621484 +0000 UTC m=+649.873552157" Oct 29 21:53:08.710930 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.710899 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:08.715876 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:08.715852 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:09.090260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:09.090178 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:09.091033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:09.091016 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:12.670703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.670673 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:12.675292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.675272 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.692193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.692173 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2976cd0d\"" Oct 29 21:53:12.703403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.703379 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2976cd0d-dockercfg-qwdjf\"" Oct 29 21:53:12.706292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.706267 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:12.767598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.767578 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:12.770884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.770864 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:12.793318 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.793262 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn22z\" (UniqueName: \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.808096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.793464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.817235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.817190 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:12.894720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.894691 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn22z\" (UniqueName: \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.894888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.894763 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:12.894888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.894786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4h7\" (UniqueName: \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:12.894888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.894811 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.895524 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.895504 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.923529 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.923475 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn22z\" (UniqueName: \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\") pod \"payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.988019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.987994 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:12.995361 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.995341 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:12.995427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.995375 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4h7\" (UniqueName: \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:12.996217 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:12.996196 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:13.020096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.020055 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4h7\" (UniqueName: \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\") pod \"payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:13.087106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.087064 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:13.288412 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.288303 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:13.291182 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:13.291154 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77539a63_a8d5_4b3b_8940_33638e038942.slice/crio-3ea3d4959b143d2682e0179805e86ba67c00d2ac19a4b89ec68ab67025d9d3d0 WatchSource:0}: Error finding container 3ea3d4959b143d2682e0179805e86ba67c00d2ac19a4b89ec68ab67025d9d3d0: Status 404 returned error can't find the container with id 3ea3d4959b143d2682e0179805e86ba67c00d2ac19a4b89ec68ab67025d9d3d0 Oct 29 21:53:13.293187 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.293170 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:53:13.401679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.401595 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:13.404529 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:13.404501 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba146607_c0f2_4b8c_a560_658dce51c606.slice/crio-8ec84ce689470b66a6f1c4b718c59b465d5e19827770076ad478864fbfea6b1c WatchSource:0}: Error finding container 8ec84ce689470b66a6f1c4b718c59b465d5e19827770076ad478864fbfea6b1c: Status 404 returned error can't find the container with id 8ec84ce689470b66a6f1c4b718c59b465d5e19827770076ad478864fbfea6b1c Oct 29 21:53:13.670456 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.670388 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:13.674264 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.674238 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.699711 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.699688 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:13.701652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.701625 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxxl\" (UniqueName: \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.701781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.701686 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.708755 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.708730 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5d56935a\"" Oct 29 21:53:13.708755 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.708750 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5d56935a-dockercfg-gwptt\"" Oct 29 21:53:13.785768 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.777243 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:13.785768 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.781900 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.798287 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.798246 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:13.807773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.807237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7xn\" (UniqueName: \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.807773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.807321 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxxl\" (UniqueName: \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.807773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.807375 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.807773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.807433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.809836 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.809787 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.833418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.833363 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxxl\" (UniqueName: \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\") pod \"payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:13.909096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.908321 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7xn\" (UniqueName: \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.909096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.908451 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.909678 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.909626 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.938782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.938716 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7xn\" (UniqueName: \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\") pod \"payload-processing-e2e-ait-5d56935a-5557d547bb-6985z\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:13.992792 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:13.992545 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:14.109488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.109464 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:14.120544 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.120459 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" event={"ID":"77539a63-a8d5-4b3b-8940-33638e038942","Type":"ContainerStarted","Data":"3ea3d4959b143d2682e0179805e86ba67c00d2ac19a4b89ec68ab67025d9d3d0"} Oct 29 21:53:14.122427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.122394 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" event={"ID":"ba146607-c0f2-4b8c-a560-658dce51c606","Type":"ContainerStarted","Data":"8ec84ce689470b66a6f1c4b718c59b465d5e19827770076ad478864fbfea6b1c"} Oct 29 21:53:14.316070 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.316033 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:14.430152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.430124 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:14.434651 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:14.434624 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d17ac01_88b7_44fe_bd98_cfa5be9053e4.slice/crio-a5fd3022d5f2fe9eaadb6673dde4a3e7c4d2187ae2210d2ef2f1997ef82b4491 WatchSource:0}: Error finding container a5fd3022d5f2fe9eaadb6673dde4a3e7c4d2187ae2210d2ef2f1997ef82b4491: Status 404 returned error can't find the container with id a5fd3022d5f2fe9eaadb6673dde4a3e7c4d2187ae2210d2ef2f1997ef82b4491 Oct 29 21:53:14.735069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.735033 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:14.740857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.740835 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:14.761604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.760730 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:14.761604 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.761551 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-044e79d4\"" Oct 29 21:53:14.767245 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.767218 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-044e79d4-dockercfg-jbj67\"" Oct 29 21:53:14.831666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.830311 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:14.844036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.844003 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:14.851892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.851514 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:14.921925 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.921895 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:14.922134 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.922118 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:14.922288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.922278 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpnp\" (UniqueName: \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:14.922356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:14.922347 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8v6\" (UniqueName: \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.023224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.023146 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpnp\" (UniqueName: \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:15.023224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.023198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8v6\" (UniqueName: \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.023449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.023279 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:15.023449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.023307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.024005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.023983 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.024956 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.024931 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:15.051640 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.050221 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpnp\" (UniqueName: \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\") pod \"payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:15.064225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.052437 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8v6\" (UniqueName: \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\") pod \"payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.077757 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.076864 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:15.134457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.133136 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" event={"ID":"d67f4d1d-4981-4405-8565-e4101c8d1262","Type":"ContainerStarted","Data":"df570136311f735df02e69dc3196270a7da8856d1af5f2f96717e3429fac82bc"} Oct 29 21:53:15.145109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.137371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" event={"ID":"4d17ac01-88b7-44fe-bd98-cfa5be9053e4","Type":"ContainerStarted","Data":"a5fd3022d5f2fe9eaadb6673dde4a3e7c4d2187ae2210d2ef2f1997ef82b4491"} Oct 29 21:53:15.188804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.187490 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:15.436601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.436527 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:15.445627 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:15.445597 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c105780_8d5a_4b8f_a360_178aaf23f6a1.slice/crio-dd45e1b98d6460d712b5fbbbd0c7c48e9ae1320045133ce167b2c2d95906055b WatchSource:0}: Error finding container dd45e1b98d6460d712b5fbbbd0c7c48e9ae1320045133ce167b2c2d95906055b: Status 404 returned error can't find the container with id dd45e1b98d6460d712b5fbbbd0c7c48e9ae1320045133ce167b2c2d95906055b Oct 29 21:53:15.522787 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:15.522760 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:16.020922 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.019092 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 21:53:16.024691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.024667 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035271 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035344 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035399 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035445 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035471 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035496 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035566 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psczm\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.037128 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.035583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.043500 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.043447 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29c2f2-openshift-default-dockercfg-t47zd\"" Oct 29 21:53:16.061193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.061165 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136360 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136417 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136468 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136588 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psczm\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136624 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136688 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.136955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.136767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.137351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.137165 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.142494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.137707 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.142494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.137909 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.142494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.138402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.142494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.141757 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.182103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.181755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psczm\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.182103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.181997 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.197375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.196474 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.199165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.198965 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.201557 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.201523 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" event={"ID":"0801c58b-90e6-4072-95d7-89c6b22efe7f","Type":"ContainerStarted","Data":"12bfe4b59b476fd9db6425d6cfa8f6a61f0cd5b4310c6c8370c9d64fcd91c280"} Oct 29 21:53:16.211375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.211347 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" event={"ID":"1c105780-8d5a-4b8f-a360-178aaf23f6a1","Type":"ContainerStarted","Data":"dd45e1b98d6460d712b5fbbbd0c7c48e9ae1320045133ce167b2c2d95906055b"} Oct 29 21:53:16.346563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.346479 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:16.690063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.689825 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 21:53:16.693696 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:16.693661 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cc10bc_3596_4c2c_b4b9_4022dfed74fb.slice/crio-2e00fc832cc3844ca543ea5a6f5428456c9380ef9ac6bfe1f87b0fde1b799a64 WatchSource:0}: Error finding container 2e00fc832cc3844ca543ea5a6f5428456c9380ef9ac6bfe1f87b0fde1b799a64: Status 404 returned error can't find the container with id 2e00fc832cc3844ca543ea5a6f5428456c9380ef9ac6bfe1f87b0fde1b799a64 Oct 29 21:53:16.697920 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.697873 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:16.698026 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.697967 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:16.698110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:16.698022 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:17.225448 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:17.225406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerStarted","Data":"01c2ba99d4a3143c59091ce324cc15cef31fddb527dad5089af466911faa3236"} Oct 29 21:53:17.225915 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:17.225478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerStarted","Data":"2e00fc832cc3844ca543ea5a6f5428456c9380ef9ac6bfe1f87b0fde1b799a64"} Oct 29 21:53:17.319950 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:17.319888 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podStartSLOduration=2.319868578 podStartE2EDuration="2.319868578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:17.302501902 +0000 UTC m=+659.044432598" watchObservedRunningTime="2026-10-29 21:53:17.319868578 +0000 UTC m=+659.061799255" Oct 29 21:53:17.363391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:17.363352 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:17.370676 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:17.370650 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:18.259904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.259201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" event={"ID":"77539a63-a8d5-4b3b-8940-33638e038942","Type":"ContainerStarted","Data":"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f"} Oct 29 21:53:18.260442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.260185 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:18.262873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.262345 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" event={"ID":"ba146607-c0f2-4b8c-a560-658dce51c606","Type":"ContainerStarted","Data":"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d"} Oct 29 21:53:18.262873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.262719 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:18.262873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.262766 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:18.263552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.263534 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:18.302348 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.302298 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" podStartSLOduration=1.897643704 podStartE2EDuration="6.302284536s" totalImagesPullingTime="4.404640832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:18.301003575 +0000 UTC m=+660.042934250" watchObservedRunningTime="2026-10-29 21:53:18.302284536 +0000 UTC m=+660.044215209" Oct 29 21:53:18.415253 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:18.415201 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" podStartSLOduration=2.449208237 podStartE2EDuration="6.415182506s" totalImagesPullingTime="3.965974269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:18.400791968 +0000 UTC m=+660.142722644" watchObservedRunningTime="2026-10-29 21:53:18.415182506 +0000 UTC m=+660.157113179" Oct 29 21:53:19.272902 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.272862 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" event={"ID":"0801c58b-90e6-4072-95d7-89c6b22efe7f","Type":"ContainerStarted","Data":"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362"} Oct 29 21:53:19.273312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.273011 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:19.274935 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.274902 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" event={"ID":"1c105780-8d5a-4b8f-a360-178aaf23f6a1","Type":"ContainerStarted","Data":"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f"} Oct 29 21:53:19.275555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.275532 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:19.277298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.277274 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" event={"ID":"4d17ac01-88b7-44fe-bd98-cfa5be9053e4","Type":"ContainerStarted","Data":"aebee2c69ae39fc29df2d3aa7ca8affb3aa7a0cd593d61bb8278e2fa61085333"} Oct 29 21:53:19.277413 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.277379 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:19.278940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.278909 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" event={"ID":"d67f4d1d-4981-4405-8565-e4101c8d1262","Type":"ContainerStarted","Data":"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9"} Oct 29 21:53:19.279418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.279401 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:19.324029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.323980 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" podStartSLOduration=2.534060609 podStartE2EDuration="5.323966244s" totalImagesPullingTime="2.789905635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:19.323055398 +0000 UTC m=+661.064986087" watchObservedRunningTime="2026-10-29 21:53:19.323966244 +0000 UTC m=+661.065896910" Oct 29 21:53:19.367889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.367840 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" podStartSLOduration=1.764794519 podStartE2EDuration="6.367826566s" totalImagesPullingTime="4.603032047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:19.36718894 +0000 UTC m=+661.109119614" watchObservedRunningTime="2026-10-29 21:53:19.367826566 +0000 UTC m=+661.109757239" Oct 29 21:53:19.412349 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.412306 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" podStartSLOduration=2.448469615 podStartE2EDuration="5.412292842s" totalImagesPullingTime="2.963823227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:19.410358884 +0000 UTC m=+661.152289556" watchObservedRunningTime="2026-10-29 21:53:19.412292842 +0000 UTC m=+661.154223516" Oct 29 21:53:19.453096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.452994 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" podStartSLOduration=1.905895246 podStartE2EDuration="6.452975793s" totalImagesPullingTime="4.547080547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:19.451396085 +0000 UTC m=+661.193326762" watchObservedRunningTime="2026-10-29 21:53:19.452975793 +0000 UTC m=+661.194906451" Oct 29 21:53:19.676732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.666112 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 21:53:19.690097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.690032 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 21:53:19.690421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.690209 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.707053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.706672 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f3b067-dockercfg-jbx7g\"" Oct 29 21:53:19.707053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.706810 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f3b067\"" Oct 29 21:53:19.799814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.799790 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.799960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.799855 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjm6\" (UniqueName: \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.900858 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.900831 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.900978 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.900887 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjm6\" (UniqueName: \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.901529 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.901511 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:19.925450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:19.925425 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjm6\" (UniqueName: \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\") pod \"payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:20.010439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:20.010415 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:20.353149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:20.347631 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 21:53:20.357339 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:20.357314 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6ea3c0_2aa9_42d2_a097_3570b9ebb0a7.slice/crio-7fc530f547159d238724fd409e9c0b5a1053dee81921b809800793487d26244c WatchSource:0}: Error finding container 7fc530f547159d238724fd409e9c0b5a1053dee81921b809800793487d26244c: Status 404 returned error can't find the container with id 7fc530f547159d238724fd409e9c0b5a1053dee81921b809800793487d26244c Oct 29 21:53:21.291798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:21.291747 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" event={"ID":"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7","Type":"ContainerStarted","Data":"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672"} Oct 29 21:53:21.292010 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:21.291842 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" event={"ID":"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7","Type":"ContainerStarted","Data":"7fc530f547159d238724fd409e9c0b5a1053dee81921b809800793487d26244c"} Oct 29 21:53:22.296723 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:22.296694 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:24.132209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:24.132178 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:53:24.132584 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:24.132226 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:53:25.822241 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.822163 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" podStartSLOduration=6.8221490540000005 podStartE2EDuration="6.822149054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:21.342353877 +0000 UTC m=+663.084284551" watchObservedRunningTime="2026-10-29 21:53:25.822149054 +0000 UTC m=+667.564079726" Oct 29 21:53:25.823039 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.823023 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 21:53:25.825532 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.825513 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.853059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.853006 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-29c2f2-dockercfg-hvpr6\"" Oct 29 21:53:25.853059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.853010 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-29c2f2\"" Oct 29 21:53:25.855487 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.855457 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 21:53:25.860889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.860866 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.861000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.860897 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrgq\" (UniqueName: \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.892907 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.892844 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 21:53:25.895358 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.895342 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:25.918226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.917442 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 21:53:25.962641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.962609 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:25.962821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.962756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.962821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.962796 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrgq\" (UniqueName: \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.962911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.962825 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9frw\" (UniqueName: \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:25.963712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.963681 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:25.987596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:25.987565 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrgq\" (UniqueName: \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\") pod \"payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:26.064056 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.064026 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9frw\" (UniqueName: \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:26.064232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.064151 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:26.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.064789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:26.088869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.088843 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9frw\" (UniqueName: \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\") pod \"payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:26.136710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.136683 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:26.208430 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.208400 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:26.325292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.325248 4543 generic.go:356] "Generic (PLEG): container finished" podID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerID="d30f2f1ba5c7f7f60fa0fbc8308736543830f2cca1548021ab10d0e0ed745426" exitCode=0 Oct 29 21:53:26.325459 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.325305 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerDied","Data":"d30f2f1ba5c7f7f60fa0fbc8308736543830f2cca1548021ab10d0e0ed745426"} Oct 29 21:53:26.325789 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.325770 4543 scope.go:122] "RemoveContainer" containerID="d30f2f1ba5c7f7f60fa0fbc8308736543830f2cca1548021ab10d0e0ed745426" Oct 29 21:53:26.327193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.327164 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:26.327261 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.327236 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:26.327298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.327268 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:26.440293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.440265 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 21:53:26.444105 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:26.444047 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f8bfba_f489_4e98_b41a_c97c8802c124.slice/crio-55ac63efc4aad46394dac54338c209e981c4c640edd90a4fe928bc59f456cb2c WatchSource:0}: Error finding container 55ac63efc4aad46394dac54338c209e981c4c640edd90a4fe928bc59f456cb2c: Status 404 returned error can't find the container with id 55ac63efc4aad46394dac54338c209e981c4c640edd90a4fe928bc59f456cb2c Oct 29 21:53:26.530914 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:26.530846 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 21:53:26.533913 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:26.533883 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790fef06_9bae_4bec_b1ae_1e67749c1295.slice/crio-b129b1aa2c33ea467db2b7ef76eeffc1a7b8d941aefc306b1145d382d63756e5 WatchSource:0}: Error finding container b129b1aa2c33ea467db2b7ef76eeffc1a7b8d941aefc306b1145d382d63756e5: Status 404 returned error can't find the container with id b129b1aa2c33ea467db2b7ef76eeffc1a7b8d941aefc306b1145d382d63756e5 Oct 29 21:53:27.355840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.355788 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerStarted","Data":"1988469ef0ba4f4ba398df61660182982f8d225bde90a723bc696c5be3898332"} Oct 29 21:53:27.357497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.357466 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" event={"ID":"b3f8bfba-f489-4e98-b41a-c97c8802c124","Type":"ContainerStarted","Data":"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9"} Oct 29 21:53:27.357643 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.357519 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" event={"ID":"b3f8bfba-f489-4e98-b41a-c97c8802c124","Type":"ContainerStarted","Data":"55ac63efc4aad46394dac54338c209e981c4c640edd90a4fe928bc59f456cb2c"} Oct 29 21:53:27.357643 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.357550 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:27.358982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.358959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" event={"ID":"790fef06-9bae-4bec-b1ae-1e67749c1295","Type":"ContainerStarted","Data":"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9"} Oct 29 21:53:27.359114 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.358997 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" event={"ID":"790fef06-9bae-4bec-b1ae-1e67749c1295","Type":"ContainerStarted","Data":"b129b1aa2c33ea467db2b7ef76eeffc1a7b8d941aefc306b1145d382d63756e5"} Oct 29 21:53:27.359114 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.359019 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:27.453216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.453179 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:27.453497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.453468 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:27.453635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.453218 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:27.479255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.479202 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" podStartSLOduration=2.479188605 podStartE2EDuration="2.479188605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:27.478397779 +0000 UTC m=+669.220328444" watchObservedRunningTime="2026-10-29 21:53:27.479188605 +0000 UTC m=+669.221119279" Oct 29 21:53:27.520160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:27.520112 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" podStartSLOduration=2.520098248 podStartE2EDuration="2.520098248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:27.518300418 +0000 UTC m=+669.260231095" watchObservedRunningTime="2026-10-29 21:53:27.520098248 +0000 UTC m=+669.262028920" Oct 29 21:53:27.988808 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:27.988760 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:53:28.451152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:28.450431 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:28.451152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:28.450483 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:29.449612 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.449521 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:29.450000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.449857 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:29.811862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.810531 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt"] Oct 29 21:53:29.813238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.813220 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:29.885237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.885203 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:53:29.889096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.885531 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:53:29.897097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.889445 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:53:29.916111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.903593 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:53:29.916570 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.912175 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtpb\" (UniqueName: \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:29.916570 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.916307 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:29.916570 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.916366 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:29.916570 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:29.914142 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt"] Oct 29 21:53:30.017783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.017731 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtpb\" (UniqueName: \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.017979 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.017827 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.017979 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.017873 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.018995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.018966 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.020482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.020458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.116125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.115154 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtpb\" (UniqueName: \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\") pod \"maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.215803 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.215766 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:30.309629 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.309597 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:30.311171 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.311150 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:30.312916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.312792 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:30.314465 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.314447 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:30.447882 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.447518 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:30.447882 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.447582 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:30.667153 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:30.665959 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt"] Oct 29 21:53:31.306101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:31.305978 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:31.308098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:31.307530 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:31.478099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:31.470335 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:31.496215 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:31.496167 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:31.516491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:31.516461 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" event={"ID":"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944","Type":"ContainerStarted","Data":"3cfbc7b0f8cfeb31522e220b0790ac9a7e3801e420fa4c13a371a434b2f2ec0e"} Oct 29 21:53:32.446568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:32.446527 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 21:53:32.451702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:32.451273 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 21:53:33.305998 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:33.305967 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 21:53:33.466102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:33.462377 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:33.466102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:33.463717 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:33.485162 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:33.485067 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:35.586292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:35.586198 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" event={"ID":"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944","Type":"ContainerStarted","Data":"c0fd0d4aa3e8771b83070fe5d859704808aafa96c1de2851eee78c976ae61725"} Oct 29 21:53:35.629387 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:35.629328 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" podStartSLOduration=2.056752553 podStartE2EDuration="6.629308396s" totalImagesPullingTime="4.572555843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:35.627555953 +0000 UTC m=+677.369486630" watchObservedRunningTime="2026-10-29 21:53:35.629308396 +0000 UTC m=+677.371239070" Oct 29 21:53:36.593478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:36.593372 4543 generic.go:356] "Generic (PLEG): container finished" podID="b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" containerID="c0fd0d4aa3e8771b83070fe5d859704808aafa96c1de2851eee78c976ae61725" exitCode=0 Oct 29 21:53:36.593478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:36.593459 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" event={"ID":"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944","Type":"ContainerDied","Data":"c0fd0d4aa3e8771b83070fe5d859704808aafa96c1de2851eee78c976ae61725"} Oct 29 21:53:37.791798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.791773 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:37.868619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.865458 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\" (UniqueName: \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\") pod \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " Oct 29 21:53:37.868619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.865517 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\") pod \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " Oct 29 21:53:37.868619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.865571 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\") pod \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\" (UID: \"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944\") " Oct 29 21:53:37.868619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.866277 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca" pod "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" (UID: "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:37.869978 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.869934 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb" pod "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" (UID: "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944"). InnerVolumeSpecName "kube-api-access-xwtpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:37.871326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.871292 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca" pod "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" (UID: "b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:37.966493 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.966461 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtpb\" (UniqueName: \"kubernetes.io/projected/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-kube-api-access-xwtpb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:37.966493 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.966490 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:37.966685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:37.966501 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:38.314474 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.313683 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:57008->10.133.0.33:15021: read: connection reset by peer" start-of-body= Oct 29 21:53:38.314474 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.313747 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:57008->10.133.0.33:15021: read: connection reset by peer" Oct 29 21:53:38.367063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.367036 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 21:53:38.367199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.367120 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 21:53:38.607236 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.607150 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" Oct 29 21:53:38.607236 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.607171 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt" event={"ID":"b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944","Type":"ContainerDied","Data":"3cfbc7b0f8cfeb31522e220b0790ac9a7e3801e420fa4c13a371a434b2f2ec0e"} Oct 29 21:53:38.607236 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.607227 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfbc7b0f8cfeb31522e220b0790ac9a7e3801e420fa4c13a371a434b2f2ec0e" Oct 29 21:53:38.608853 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.608826 4543 generic.go:356] "Generic (PLEG): container finished" podID="e81b469b-69ce-4be7-8093-fec68c3365af" containerID="9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3" exitCode=0 Oct 29 21:53:38.608993 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.608867 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerDied","Data":"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3"} Oct 29 21:53:38.609229 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.609211 4543 scope.go:122] "RemoveContainer" containerID="9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3" Oct 29 21:53:38.610793 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.610755 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:38.610890 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.610829 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:38.610890 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:38.610865 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:39.091806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.091777 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:39.092194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.091838 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:39.128104 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.127364 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l"] Oct 29 21:53:39.128321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.128250 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" containerName="revoke-keys" Oct 29 21:53:39.128321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.128273 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" containerName="revoke-keys" Oct 29 21:53:39.128451 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.128410 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" containerName="revoke-keys" Oct 29 21:53:39.137099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.133962 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.156213 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.152449 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:53:39.156213 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.155985 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l"] Oct 29 21:53:39.157375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.157202 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:53:39.157375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.157281 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:53:39.157375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.157343 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:53:39.281100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.280698 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98h6\" (UniqueName: \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.281100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.280894 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.281100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.280926 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.382437 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.381937 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98h6\" (UniqueName: \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.382437 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.382169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.383018 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.382987 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.383232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.382202 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.384918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.384895 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.414086 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.414051 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98h6\" (UniqueName: \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\") pod \"maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.506099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.505777 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:39.624231 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.622603 4543 generic.go:356] "Generic (PLEG): container finished" podID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerID="c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd" exitCode=0 Oct 29 21:53:39.624231 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.622814 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerDied","Data":"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd"} Oct 29 21:53:39.624231 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.623308 4543 scope.go:122] "RemoveContainer" containerID="c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd" Oct 29 21:53:39.625517 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.625481 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:39.625632 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.625559 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:39.625632 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.625599 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:39.627872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.627848 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerStarted","Data":"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c"} Oct 29 21:53:39.890129 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.887694 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l"] Oct 29 21:53:39.893720 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:39.893555 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c1af36_3f57_420d_943a_65915e17f337.slice/crio-aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2 WatchSource:0}: Error finding container aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2: Status 404 returned error can't find the container with id aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2 Oct 29 21:53:39.984815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.984472 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:39.984815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.984568 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:53:39.984815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.984613 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 21:53:39.987966 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:39.987628 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:40.046766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.046134 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:40.461109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.459860 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t"] Oct 29 21:53:40.501554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.501500 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t"] Oct 29 21:53:40.515246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.504225 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.618152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.617900 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.618152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.618111 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.618406 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.618237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jrt\" (UniqueName: \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.715512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.714526 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerStarted","Data":"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d"} Oct 29 21:53:40.719175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.718700 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" event={"ID":"b4c1af36-3f57-420d-943a-65915e17f337","Type":"ContainerStarted","Data":"3335516cd11101deaad0efbe8c32d99e61f50f04472de8264bc8088b98ff26b4"} Oct 29 21:53:40.719175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.718755 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" event={"ID":"b4c1af36-3f57-420d-943a-65915e17f337","Type":"ContainerStarted","Data":"aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2"} Oct 29 21:53:40.720313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.720287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.823097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.821859 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jrt\" (UniqueName: \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.823097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.822053 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.823821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.823449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.926782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.926697 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:40.991100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.989220 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:53:40.991100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:40.989286 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 21:53:41.018549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.015297 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" podStartSLOduration=2.015275116 podStartE2EDuration="2.015275116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:41.010854277 +0000 UTC m=+682.752784951" watchObservedRunningTime="2026-10-29 21:53:41.015275116 +0000 UTC m=+682.757205790" Oct 29 21:53:41.020193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.020162 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jrt\" (UniqueName: \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\") pod \"maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:41.236186 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.235894 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:41.713280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.713246 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:41.713851 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.713800 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:41.713970 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.713853 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:41.725161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.724407 4543 generic.go:356] "Generic (PLEG): container finished" podID="b4c1af36-3f57-420d-943a-65915e17f337" containerID="3335516cd11101deaad0efbe8c32d99e61f50f04472de8264bc8088b98ff26b4" exitCode=0 Oct 29 21:53:41.725161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.725003 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" event={"ID":"b4c1af36-3f57-420d-943a-65915e17f337","Type":"ContainerDied","Data":"3335516cd11101deaad0efbe8c32d99e61f50f04472de8264bc8088b98ff26b4"} Oct 29 21:53:41.986457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.986417 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:53:41.986723 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:41.986693 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 21:53:42.059415 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.050401 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t"] Oct 29 21:53:42.082751 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:42.082698 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d11b2a_3df4_468d_a88c_17d848f8b764.slice/crio-cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81 WatchSource:0}: Error finding container cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81: Status 404 returned error can't find the container with id cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81 Oct 29 21:53:42.711753 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.711714 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:42.711944 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.711765 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:42.743417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.743385 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" event={"ID":"c9d11b2a-3df4-468d-a88c-17d848f8b764","Type":"ContainerStarted","Data":"9e66719dad4f97058e7f3440c9b3a55aa65777c39ae77195d312ff8c2383af7f"} Oct 29 21:53:42.743981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.743948 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" event={"ID":"c9d11b2a-3df4-468d-a88c-17d848f8b764","Type":"ContainerStarted","Data":"cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81"} Oct 29 21:53:42.986963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.986455 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:53:42.986963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:42.986530 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 21:53:43.086035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.086005 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:43.126917 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.126496 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" podStartSLOduration=3.126478369 podStartE2EDuration="3.126478369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:42.807307568 +0000 UTC m=+684.549238242" watchObservedRunningTime="2026-10-29 21:53:43.126478369 +0000 UTC m=+684.868409043" Oct 29 21:53:43.184123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.180984 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\" (UniqueName: \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\") pod \"b4c1af36-3f57-420d-943a-65915e17f337\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " Oct 29 21:53:43.184123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.181128 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\") pod \"b4c1af36-3f57-420d-943a-65915e17f337\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " Oct 29 21:53:43.184123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.181207 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\") pod \"b4c1af36-3f57-420d-943a-65915e17f337\" (UID: \"b4c1af36-3f57-420d-943a-65915e17f337\") " Oct 29 21:53:43.184123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.181861 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca" pod "b4c1af36-3f57-420d-943a-65915e17f337" (UID: "b4c1af36-3f57-420d-943a-65915e17f337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:43.187703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.187482 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6" pod "b4c1af36-3f57-420d-943a-65915e17f337" (UID: "b4c1af36-3f57-420d-943a-65915e17f337"). InnerVolumeSpecName "kube-api-access-w98h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:43.187703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.187671 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca" pod "b4c1af36-3f57-420d-943a-65915e17f337" (UID: "b4c1af36-3f57-420d-943a-65915e17f337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:43.285814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.285763 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.285814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.285807 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c1af36-3f57-420d-943a-65915e17f337-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.286048 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.285824 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98h6\" (UniqueName: \"kubernetes.io/projected/b4c1af36-3f57-420d-943a-65915e17f337-kube-api-access-w98h6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.711521 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.711483 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:43.711693 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.711546 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:43.778110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.773679 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" event={"ID":"b4c1af36-3f57-420d-943a-65915e17f337","Type":"ContainerDied","Data":"aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2"} Oct 29 21:53:43.778110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.773742 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab72c46b640ed73d8bb8cadbcf4d6ea03bcab33fdef1224c4c168df0a627ef2" Oct 29 21:53:43.778110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.773818 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l" Oct 29 21:53:43.778110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.777208 4543 generic.go:356] "Generic (PLEG): container finished" podID="c9d11b2a-3df4-468d-a88c-17d848f8b764" containerID="9e66719dad4f97058e7f3440c9b3a55aa65777c39ae77195d312ff8c2383af7f" exitCode=0 Oct 29 21:53:43.778110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.777377 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" event={"ID":"c9d11b2a-3df4-468d-a88c-17d848f8b764","Type":"ContainerDied","Data":"9e66719dad4f97058e7f3440c9b3a55aa65777c39ae77195d312ff8c2383af7f"} Oct 29 21:53:43.983841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.983753 4543 patch_prober.go:29] interesting pod/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 21:53:43.983841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:43.983828 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 21:53:44.688959 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.688907 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:44.689746 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.689715 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" containerID="cri-o://1988469ef0ba4f4ba398df61660182982f8d225bde90a723bc696c5be3898332" gracePeriod=30 Oct 29 21:53:44.711822 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.711790 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:44.711941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.711849 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:44.988884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.988841 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" podUID="0801c58b-90e6-4072-95d7-89c6b22efe7f" containerName="payload-processing" containerID="cri-o://07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362" gracePeriod=25 Oct 29 21:53:44.992574 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:44.992501 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:45.007939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.007859 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\") pod \"c9d11b2a-3df4-468d-a88c-17d848f8b764\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " Oct 29 21:53:45.007939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.007914 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\") pod \"c9d11b2a-3df4-468d-a88c-17d848f8b764\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " Oct 29 21:53:45.008381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.007987 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\" (UniqueName: \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\") pod \"c9d11b2a-3df4-468d-a88c-17d848f8b764\" (UID: \"c9d11b2a-3df4-468d-a88c-17d848f8b764\") " Oct 29 21:53:45.008796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.008764 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca" pod "c9d11b2a-3df4-468d-a88c-17d848f8b764" (UID: "c9d11b2a-3df4-468d-a88c-17d848f8b764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:45.010668 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.010641 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt" pod "c9d11b2a-3df4-468d-a88c-17d848f8b764" (UID: "c9d11b2a-3df4-468d-a88c-17d848f8b764"). InnerVolumeSpecName "kube-api-access-m2jrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:45.010928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.010910 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca" pod "c9d11b2a-3df4-468d-a88c-17d848f8b764" (UID: "c9d11b2a-3df4-468d-a88c-17d848f8b764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:45.075102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.068671 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" podUID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" containerName="payload-pre-processing" containerID="cri-o://79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f" gracePeriod=25 Oct 29 21:53:45.109850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.109817 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.109850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.109848 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d11b2a-3df4-468d-a88c-17d848f8b764-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.109850 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.109858 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jrt\" (UniqueName: \"kubernetes.io/projected/c9d11b2a-3df4-468d-a88c-17d848f8b764-kube-api-access-m2jrt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.198913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.198883 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:45.199473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.199361 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:45.200558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.200493 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:53:45.381442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.381418 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:45.416384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.416351 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\") pod \"0801c58b-90e6-4072-95d7-89c6b22efe7f\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " Oct 29 21:53:45.416518 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.416420 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\" (UniqueName: \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\") pod \"0801c58b-90e6-4072-95d7-89c6b22efe7f\" (UID: \"0801c58b-90e6-4072-95d7-89c6b22efe7f\") " Oct 29 21:53:45.417910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.417882 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume" pod "0801c58b-90e6-4072-95d7-89c6b22efe7f" (UID: "0801c58b-90e6-4072-95d7-89c6b22efe7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:45.420207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.420181 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6" pod "0801c58b-90e6-4072-95d7-89c6b22efe7f" (UID: "0801c58b-90e6-4072-95d7-89c6b22efe7f"). InnerVolumeSpecName "kube-api-access-5g8v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:45.468739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.468711 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:45.517952 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.517926 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\" (UniqueName: \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\") pod \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " Oct 29 21:53:45.518107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.517988 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\") pod \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\" (UID: \"1c105780-8d5a-4b8f-a360-178aaf23f6a1\") " Oct 29 21:53:45.518445 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.518425 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8v6\" (UniqueName: \"kubernetes.io/projected/0801c58b-90e6-4072-95d7-89c6b22efe7f-kube-api-access-5g8v6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.518528 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.518448 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801c58b-90e6-4072-95d7-89c6b22efe7f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.518528 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.518424 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume" pod "1c105780-8d5a-4b8f-a360-178aaf23f6a1" (UID: "1c105780-8d5a-4b8f-a360-178aaf23f6a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:45.520291 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.520266 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp" pod "1c105780-8d5a-4b8f-a360-178aaf23f6a1" (UID: "1c105780-8d5a-4b8f-a360-178aaf23f6a1"). InnerVolumeSpecName "kube-api-access-xdpnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:45.619766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.619702 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpnp\" (UniqueName: \"kubernetes.io/projected/1c105780-8d5a-4b8f-a360-178aaf23f6a1-kube-api-access-xdpnp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.619766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.619725 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c105780-8d5a-4b8f-a360-178aaf23f6a1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:45.711053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.711012 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 21:53:45.711226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.711091 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 21:53:45.792811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.792783 4543 generic.go:356] "Generic (PLEG): container finished" podID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" containerID="79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f" exitCode=0 Oct 29 21:53:45.792995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.792852 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" Oct 29 21:53:45.792995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.792880 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" event={"ID":"1c105780-8d5a-4b8f-a360-178aaf23f6a1","Type":"ContainerDied","Data":"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f"} Oct 29 21:53:45.792995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.792945 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp" event={"ID":"1c105780-8d5a-4b8f-a360-178aaf23f6a1","Type":"ContainerDied","Data":"dd45e1b98d6460d712b5fbbbd0c7c48e9ae1320045133ce167b2c2d95906055b"} Oct 29 21:53:45.792995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.792986 4543 scope.go:122] "RemoveContainer" containerID="79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f" Oct 29 21:53:45.794593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.794440 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" event={"ID":"c9d11b2a-3df4-468d-a88c-17d848f8b764","Type":"ContainerDied","Data":"cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81"} Oct 29 21:53:45.794593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.794481 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab98da7ef6cac4561bf8570f60fd853adf40adaf2e15d73cc38255040b3be81" Oct 29 21:53:45.794593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.794522 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t" Oct 29 21:53:45.796123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.796041 4543 generic.go:356] "Generic (PLEG): container finished" podID="0801c58b-90e6-4072-95d7-89c6b22efe7f" containerID="07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362" exitCode=0 Oct 29 21:53:45.796123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.796107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" event={"ID":"0801c58b-90e6-4072-95d7-89c6b22efe7f","Type":"ContainerDied","Data":"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362"} Oct 29 21:53:45.796276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.796149 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" event={"ID":"0801c58b-90e6-4072-95d7-89c6b22efe7f","Type":"ContainerDied","Data":"12bfe4b59b476fd9db6425d6cfa8f6a61f0cd5b4310c6c8370c9d64fcd91c280"} Oct 29 21:53:45.796276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.796160 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69" Oct 29 21:53:45.810233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.810214 4543 scope.go:122] "RemoveContainer" containerID="79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f" Oct 29 21:53:45.810580 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:45.810555 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f\": container with ID starting with 79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f not found: ID does not exist" containerID="79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f" Oct 29 21:53:45.810664 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.810582 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f"} err="failed to get container status \"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f\": rpc error: code = NotFound desc = could not find container \"79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f\": container with ID starting with 79ee1951b1ac6e6f1ee4cd44b40ed6b2d9d00f628539bc1cb6d842552112450f not found: ID does not exist" Oct 29 21:53:45.810664 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.810598 4543 scope.go:122] "RemoveContainer" containerID="07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362" Oct 29 21:53:45.821208 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.821186 4543 scope.go:122] "RemoveContainer" containerID="07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362" Oct 29 21:53:45.821480 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:45.821457 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362\": container with ID starting with 07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362 not found: ID does not exist" containerID="07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362" Oct 29 21:53:45.821567 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.821490 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362"} err="failed to get container status \"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362\": rpc error: code = NotFound desc = could not find container \"07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362\": container with ID starting with 07afb80f01e2b5050c78e216b2faf2323f78780f4f86909c20083cc9c5e69362 not found: ID does not exist" Oct 29 21:53:45.855759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.855725 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:45.859575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.859540 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-044e79d4-5fd64cc787-x87gp"] Oct 29 21:53:45.912864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.912806 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:45.918174 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.918151 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-044e79d4-78f68dc86d-fkw69"] Oct 29 21:53:45.963207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963175 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 21:53:45.963816 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963796 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b4c1af36-3f57-420d-943a-65915e17f337" containerName="revoke-keys" Oct 29 21:53:45.963816 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963815 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c1af36-3f57-420d-943a-65915e17f337" containerName="revoke-keys" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963841 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" containerName="payload-pre-processing" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963849 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" containerName="payload-pre-processing" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963867 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0801c58b-90e6-4072-95d7-89c6b22efe7f" containerName="payload-processing" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963875 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0801c58b-90e6-4072-95d7-89c6b22efe7f" containerName="payload-processing" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963891 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c9d11b2a-3df4-468d-a88c-17d848f8b764" containerName="revoke-keys" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963900 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d11b2a-3df4-468d-a88c-17d848f8b764" containerName="revoke-keys" Oct 29 21:53:45.964005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.963999 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c1af36-3f57-420d-943a-65915e17f337" containerName="revoke-keys" Oct 29 21:53:45.964373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.964012 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" containerName="payload-pre-processing" Oct 29 21:53:45.964373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.964021 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0801c58b-90e6-4072-95d7-89c6b22efe7f" containerName="payload-processing" Oct 29 21:53:45.964373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.964030 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d11b2a-3df4-468d-a88c-17d848f8b764" containerName="revoke-keys" Oct 29 21:53:45.981390 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.978298 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:45.999234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.997173 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 21:53:45.999234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:45.997367 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-91520c-openshift-default-dockercfg-gg8xj\"" Oct 29 21:53:46.023557 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023526 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023692 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023575 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023779 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023779 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023727 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023885 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023780 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023885 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023827 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2gl\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023990 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023887 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.023990 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.023953 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.024124 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.024019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.124820 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2gl\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.124913 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.124997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125068 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125130 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125162 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125224 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.125849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.125281 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.127114 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.127038 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.127438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.127388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.128359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.128334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.128497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.128402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.128675 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.128651 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.133621 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.132455 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.134692 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.134580 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.156037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.152885 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2gl\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.156736 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.156674 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.306462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.306429 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:46.388556 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.388526 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0801c58b-90e6-4072-95d7-89c6b22efe7f" path="/var/lib/kubelet/pods/0801c58b-90e6-4072-95d7-89c6b22efe7f/volumes" Oct 29 21:53:46.389298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.389276 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c105780-8d5a-4b8f-a360-178aaf23f6a1" path="/var/lib/kubelet/pods/1c105780-8d5a-4b8f-a360-178aaf23f6a1/volumes" Oct 29 21:53:46.627625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.627602 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 21:53:46.629347 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:46.629327 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9d89d4_3e13_4eb6_ac8c_6861c936cfaa.slice/crio-96d7dccf95eba0f7d4c3b52fda731dfcd3bf5bff8e4eeb373d7a36a0b705f421 WatchSource:0}: Error finding container 96d7dccf95eba0f7d4c3b52fda731dfcd3bf5bff8e4eeb373d7a36a0b705f421: Status 404 returned error can't find the container with id 96d7dccf95eba0f7d4c3b52fda731dfcd3bf5bff8e4eeb373d7a36a0b705f421 Oct 29 21:53:46.631564 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.631534 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:46.631634 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.631592 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:46.631634 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.631621 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:46.715709 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.715684 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:46.716045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.716028 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:46.716834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.716817 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:53:46.806660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.806627 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerStarted","Data":"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600"} Oct 29 21:53:46.806901 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.806877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerStarted","Data":"96d7dccf95eba0f7d4c3b52fda731dfcd3bf5bff8e4eeb373d7a36a0b705f421"} Oct 29 21:53:46.873104 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:46.870960 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podStartSLOduration=1.870939033 podStartE2EDuration="1.870939033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:46.859299068 +0000 UTC m=+688.601229766" watchObservedRunningTime="2026-10-29 21:53:46.870939033 +0000 UTC m=+688.612869707" Oct 29 21:53:47.307153 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.307116 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:47.312368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.312347 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:47.816619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.816586 4543 generic.go:356] "Generic (PLEG): container finished" podID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerID="01c2ba99d4a3143c59091ce324cc15cef31fddb527dad5089af466911faa3236" exitCode=0 Oct 29 21:53:47.816802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.816662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerDied","Data":"01c2ba99d4a3143c59091ce324cc15cef31fddb527dad5089af466911faa3236"} Oct 29 21:53:47.817207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.817141 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:47.817383 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.817365 4543 scope.go:122] "RemoveContainer" containerID="01c2ba99d4a3143c59091ce324cc15cef31fddb527dad5089af466911faa3236" Oct 29 21:53:47.818178 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.817858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:53:47.818579 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.818549 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:47.818659 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.818639 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:47.818720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:47.818690 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:53:48.263341 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:48.263295 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:48.466234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:48.466191 4543 patch_prober.go:29] interesting pod/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 21:53:48.466630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:48.466249 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 21:53:48.824330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:48.824289 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerStarted","Data":"c493474be9457120ebccea22ece55e80bad0a7cad7a11ac157167286b224f06b"} Oct 29 21:53:49.351416 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.351385 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:49.353362 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.353333 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 21:53:49.353502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.353389 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 21:53:49.858559 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.858405 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:49.859246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.859219 4543 generic.go:356] "Generic (PLEG): container finished" podID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerID="1988469ef0ba4f4ba398df61660182982f8d225bde90a723bc696c5be3898332" exitCode=0 Oct 29 21:53:49.859389 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.859363 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerDied","Data":"1988469ef0ba4f4ba398df61660182982f8d225bde90a723bc696c5be3898332"} Oct 29 21:53:49.859534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.859522 4543 scope.go:122] "RemoveContainer" containerID="d30f2f1ba5c7f7f60fa0fbc8308736543830f2cca1548021ab10d0e0ed745426" Oct 29 21:53:49.890852 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:49.890732 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:50.013137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.013111 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:50.105625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.105593 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.105884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.105863 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106022 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106228 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106404 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106388 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106569 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106550 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106683 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106833 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106818 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.106977 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106961 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\") pod \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\" (UID: \"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d\") " Oct 29 21:53:50.107301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:50.107301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106607 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:50.107301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.106820 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:50.107750 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.107722 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:50.107900 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.107736 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.107978 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.107919 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.108033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.107975 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.109261 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.109229 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:50.110313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.110267 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:50.110992 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.110967 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:53:50.111219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.111198 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 21:53:50.111301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.111270 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt" pod "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" (UID: "46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d"). InnerVolumeSpecName "kube-api-access-k78tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:50.208650 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208621 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.208650 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208651 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78tt\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-kube-api-access-k78tt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.208829 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208662 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.208829 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208678 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.208829 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208686 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.208829 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.208694 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:50.353246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.353165 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 21:53:50.353246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.353230 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 21:53:50.866731 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.866705 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" Oct 29 21:53:50.867211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.866753 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt" event={"ID":"46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d","Type":"ContainerDied","Data":"714c2685d723cfaf6695fb5cbfc180a9e6b676709f00dbcaabf79144a4cf425a"} Oct 29 21:53:50.867211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.866816 4543 scope.go:122] "RemoveContainer" containerID="1988469ef0ba4f4ba398df61660182982f8d225bde90a723bc696c5be3898332" Oct 29 21:53:50.954132 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.954065 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:50.958508 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:50.958484 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-229ab2ac-openshift-default-678dc54db5-54vdt"] Oct 29 21:53:51.353442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.353403 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 21:53:51.353630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.353460 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 21:53:51.454798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.454757 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 21:53:51.455196 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455181 4543 cpu_manager.go:414] "containerMap: removing container" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.455196 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455196 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.455293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455213 4543 cpu_manager.go:414] "containerMap: removing container" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.455293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455218 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.455359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455298 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.455359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.455307 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" containerName="istio-proxy" Oct 29 21:53:51.458440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.458418 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.471759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.471739 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 21:53:51.474513 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.474496 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-91520c\"" Oct 29 21:53:51.474857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.474836 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-91520c-dockercfg-fl6xt\"" Oct 29 21:53:51.522837 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.522795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.522967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.522915 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftd2k\" (UniqueName: \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.584060 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.583980 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 21:53:51.590041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.590014 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 21:53:51.590368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.590350 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.624276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.624220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.624416 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.624295 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftd2k\" (UniqueName: \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.624871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.624852 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.656859 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.656824 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftd2k\" (UniqueName: \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\") pod \"payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.724809 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.724777 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6t77\" (UniqueName: \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.724960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.724852 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.770762 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.770734 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:51.786456 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.786432 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:51.836948 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.836634 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6t77\" (UniqueName: \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.837549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.837523 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.840145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.839960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.844805 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.844784 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:51.888777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.888746 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6t77\" (UniqueName: \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\") pod \"payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:51.910325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:51.910126 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:52.123152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.123124 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 21:53:52.126137 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:53:52.126060 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04497cd_0a50_440b_a735_3d63585b76f5.slice/crio-2a6af3470058eadf003a719eb591c6089c0e08286d57c47f34f3a1d11d803e6a WatchSource:0}: Error finding container 2a6af3470058eadf003a719eb591c6089c0e08286d57c47f34f3a1d11d803e6a: Status 404 returned error can't find the container with id 2a6af3470058eadf003a719eb591c6089c0e08286d57c47f34f3a1d11d803e6a Oct 29 21:53:52.265160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.265131 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 21:53:52.354459 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.354424 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 21:53:52.354605 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.354484 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 21:53:52.385239 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.385185 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d" path="/var/lib/kubelet/pods/46baa5cc-9e6b-488d-b81d-cbf90eb0ae0d/volumes" Oct 29 21:53:52.568877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.568839 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt"] Oct 29 21:53:52.571164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.571140 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-044e79d4-bd9jt"] Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.888519 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" event={"ID":"0433cc1f-5fe7-4f71-9726-2c36f02abcc0","Type":"ContainerStarted","Data":"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c"} Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.888597 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" event={"ID":"0433cc1f-5fe7-4f71-9726-2c36f02abcc0","Type":"ContainerStarted","Data":"cd508aa1dc2a228088a27bdce8c03d1ca9f45dbe8b5773409fd186206d4b1262"} Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.889474 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.891513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" event={"ID":"f04497cd-0a50-440b-a735-3d63585b76f5","Type":"ContainerStarted","Data":"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a"} Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.891560 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" event={"ID":"f04497cd-0a50-440b-a735-3d63585b76f5","Type":"ContainerStarted","Data":"2a6af3470058eadf003a719eb591c6089c0e08286d57c47f34f3a1d11d803e6a"} Oct 29 21:53:52.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.891815 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:53:52.998908 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.998859 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" podStartSLOduration=1.9988450979999999 podStartE2EDuration="1.998845098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:52.996689457 +0000 UTC m=+694.738620129" watchObservedRunningTime="2026-10-29 21:53:52.998845098 +0000 UTC m=+694.740775771" Oct 29 21:53:52.999088 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:52.998940 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" podStartSLOduration=1.998936145 podStartE2EDuration="1.998936145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:52.95422205 +0000 UTC m=+694.696152725" watchObservedRunningTime="2026-10-29 21:53:52.998936145 +0000 UTC m=+694.740866817" Oct 29 21:53:53.353397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:53.353361 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 21:53:53.353566 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:53.353422 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 21:53:54.351965 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.351937 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:54.352421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.352361 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:54.352960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.352940 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 21:53:54.384716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.384688 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944" path="/var/lib/kubelet/pods/b6e41f7d-1ac2-4ddd-a597-f5d2d7a85944/volumes" Oct 29 21:53:54.859053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.859015 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" podUID="ba146607-c0f2-4b8c-a560-658dce51c606" containerName="payload-processing" containerID="cri-o://2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d" gracePeriod=25 Oct 29 21:53:54.891095 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:54.891031 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" podUID="77539a63-a8d5-4b3b-8940-33638e038942" containerName="payload-pre-processing" containerID="cri-o://ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f" gracePeriod=25 Oct 29 21:53:55.149960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.149938 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:55.178152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.178131 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:55.307059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307031 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\" (UniqueName: \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\") pod \"ba146607-c0f2-4b8c-a560-658dce51c606\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " Oct 29 21:53:55.307209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307099 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\" (UniqueName: \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\") pod \"77539a63-a8d5-4b3b-8940-33638e038942\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " Oct 29 21:53:55.307209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307137 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\") pod \"77539a63-a8d5-4b3b-8940-33638e038942\" (UID: \"77539a63-a8d5-4b3b-8940-33638e038942\") " Oct 29 21:53:55.307209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307175 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\") pod \"ba146607-c0f2-4b8c-a560-658dce51c606\" (UID: \"ba146607-c0f2-4b8c-a560-658dce51c606\") " Oct 29 21:53:55.307551 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307522 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume" pod "77539a63-a8d5-4b3b-8940-33638e038942" (UID: "77539a63-a8d5-4b3b-8940-33638e038942"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:55.307679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307522 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume" pod "ba146607-c0f2-4b8c-a560-658dce51c606" (UID: "ba146607-c0f2-4b8c-a560-658dce51c606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:55.307679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307618 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77539a63-a8d5-4b3b-8940-33638e038942-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:55.307679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.307638 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba146607-c0f2-4b8c-a560-658dce51c606-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:55.309200 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.309176 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7" pod "ba146607-c0f2-4b8c-a560-658dce51c606" (UID: "ba146607-c0f2-4b8c-a560-658dce51c606"). InnerVolumeSpecName "kube-api-access-4j4h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:55.309282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.309252 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z" pod "77539a63-a8d5-4b3b-8940-33638e038942" (UID: "77539a63-a8d5-4b3b-8940-33638e038942"). InnerVolumeSpecName "kube-api-access-rn22z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:55.408579 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.408525 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4h7\" (UniqueName: \"kubernetes.io/projected/ba146607-c0f2-4b8c-a560-658dce51c606-kube-api-access-4j4h7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:55.408579 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.408549 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn22z\" (UniqueName: \"kubernetes.io/projected/77539a63-a8d5-4b3b-8940-33638e038942-kube-api-access-rn22z\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:55.912554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.912526 4543 generic.go:356] "Generic (PLEG): container finished" podID="ba146607-c0f2-4b8c-a560-658dce51c606" containerID="2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d" exitCode=0 Oct 29 21:53:55.912687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.912575 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" Oct 29 21:53:55.912687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.912611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" event={"ID":"ba146607-c0f2-4b8c-a560-658dce51c606","Type":"ContainerDied","Data":"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d"} Oct 29 21:53:55.912687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.912669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq" event={"ID":"ba146607-c0f2-4b8c-a560-658dce51c606","Type":"ContainerDied","Data":"8ec84ce689470b66a6f1c4b718c59b465d5e19827770076ad478864fbfea6b1c"} Oct 29 21:53:55.912814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.912696 4543 scope.go:122] "RemoveContainer" containerID="2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d" Oct 29 21:53:55.914105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.914057 4543 generic.go:356] "Generic (PLEG): container finished" podID="77539a63-a8d5-4b3b-8940-33638e038942" containerID="ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f" exitCode=0 Oct 29 21:53:55.914190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.914116 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" event={"ID":"77539a63-a8d5-4b3b-8940-33638e038942","Type":"ContainerDied","Data":"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f"} Oct 29 21:53:55.914190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.914132 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" Oct 29 21:53:55.914190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.914169 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7" event={"ID":"77539a63-a8d5-4b3b-8940-33638e038942","Type":"ContainerDied","Data":"3ea3d4959b143d2682e0179805e86ba67c00d2ac19a4b89ec68ab67025d9d3d0"} Oct 29 21:53:55.922471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.922448 4543 scope.go:122] "RemoveContainer" containerID="2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d" Oct 29 21:53:55.922755 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:55.922737 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d\": container with ID starting with 2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d not found: ID does not exist" containerID="2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d" Oct 29 21:53:55.922824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.922762 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d"} err="failed to get container status \"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d\": rpc error: code = NotFound desc = could not find container \"2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d\": container with ID starting with 2c302b65e149b007eb94ceae7e05307ff890044cad416383656167a179be8a2d not found: ID does not exist" Oct 29 21:53:55.922824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.922777 4543 scope.go:122] "RemoveContainer" containerID="ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f" Oct 29 21:53:55.932482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.932467 4543 scope.go:122] "RemoveContainer" containerID="ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f" Oct 29 21:53:55.932798 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:55.932779 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f\": container with ID starting with ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f not found: ID does not exist" containerID="ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f" Oct 29 21:53:55.932881 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.932804 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f"} err="failed to get container status \"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f\": rpc error: code = NotFound desc = could not find container \"ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f\": container with ID starting with ffe5867b751d11a550219a211fe52828e8f4199b3e4d2fa7f73e5e76140ed19f not found: ID does not exist" Oct 29 21:53:55.968203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.968182 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:55.975725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:55.975704 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2976cd0d-7cd89c9c5-z9zd7"] Oct 29 21:53:56.011328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.011310 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:56.016041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.016019 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2976cd0d-78b8f485c8-9qtlq"] Oct 29 21:53:56.385857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.385833 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77539a63-a8d5-4b3b-8940-33638e038942" path="/var/lib/kubelet/pods/77539a63-a8d5-4b3b-8940-33638e038942/volumes" Oct 29 21:53:56.386229 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.386214 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba146607-c0f2-4b8c-a560-658dce51c606" path="/var/lib/kubelet/pods/ba146607-c0f2-4b8c-a560-658dce51c606/volumes" Oct 29 21:53:56.787357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.787319 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" podUID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" containerName="payload-processing" containerID="cri-o://aebee2c69ae39fc29df2d3aa7ca8affb3aa7a0cd593d61bb8278e2fa61085333" gracePeriod=25 Oct 29 21:53:56.846247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.846126 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" podUID="d67f4d1d-4981-4405-8565-e4101c8d1262" containerName="payload-pre-processing" containerID="cri-o://f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9" gracePeriod=25 Oct 29 21:53:56.921783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.921752 4543 generic.go:356] "Generic (PLEG): container finished" podID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" containerID="aebee2c69ae39fc29df2d3aa7ca8affb3aa7a0cd593d61bb8278e2fa61085333" exitCode=0 Oct 29 21:53:56.921955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:56.921829 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" event={"ID":"4d17ac01-88b7-44fe-bd98-cfa5be9053e4","Type":"ContainerDied","Data":"aebee2c69ae39fc29df2d3aa7ca8affb3aa7a0cd593d61bb8278e2fa61085333"} Oct 29 21:53:57.146436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.146409 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:57.165976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.165951 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:57.225182 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.225159 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\" (UniqueName: \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\") pod \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " Oct 29 21:53:57.225311 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.225297 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\") pod \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\" (UID: \"4d17ac01-88b7-44fe-bd98-cfa5be9053e4\") " Oct 29 21:53:57.225629 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.225607 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume" pod "4d17ac01-88b7-44fe-bd98-cfa5be9053e4" (UID: "4d17ac01-88b7-44fe-bd98-cfa5be9053e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:57.227357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.227335 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn" pod "4d17ac01-88b7-44fe-bd98-cfa5be9053e4" (UID: "4d17ac01-88b7-44fe-bd98-cfa5be9053e4"). InnerVolumeSpecName "kube-api-access-4r7xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:57.325929 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.325855 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\") pod \"d67f4d1d-4981-4405-8565-e4101c8d1262\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " Oct 29 21:53:57.326051 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.325935 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\" (UniqueName: \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\") pod \"d67f4d1d-4981-4405-8565-e4101c8d1262\" (UID: \"d67f4d1d-4981-4405-8565-e4101c8d1262\") " Oct 29 21:53:57.326246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.326222 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume" pod "d67f4d1d-4981-4405-8565-e4101c8d1262" (UID: "d67f4d1d-4981-4405-8565-e4101c8d1262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:57.326329 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.326316 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:57.326371 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.326331 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7xn\" (UniqueName: \"kubernetes.io/projected/4d17ac01-88b7-44fe-bd98-cfa5be9053e4-kube-api-access-4r7xn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:57.326371 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.326342 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67f4d1d-4981-4405-8565-e4101c8d1262-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:57.328173 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.328147 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl" pod "d67f4d1d-4981-4405-8565-e4101c8d1262" (UID: "d67f4d1d-4981-4405-8565-e4101c8d1262"). InnerVolumeSpecName "kube-api-access-8sxxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:57.427438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.427416 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxxl\" (UniqueName: \"kubernetes.io/projected/d67f4d1d-4981-4405-8565-e4101c8d1262-kube-api-access-8sxxl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:53:57.932433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.932409 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" Oct 29 21:53:57.932831 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.932431 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z" event={"ID":"4d17ac01-88b7-44fe-bd98-cfa5be9053e4","Type":"ContainerDied","Data":"a5fd3022d5f2fe9eaadb6673dde4a3e7c4d2187ae2210d2ef2f1997ef82b4491"} Oct 29 21:53:57.932831 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.932491 4543 scope.go:122] "RemoveContainer" containerID="aebee2c69ae39fc29df2d3aa7ca8affb3aa7a0cd593d61bb8278e2fa61085333" Oct 29 21:53:57.933862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.933836 4543 generic.go:356] "Generic (PLEG): container finished" podID="d67f4d1d-4981-4405-8565-e4101c8d1262" containerID="f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9" exitCode=0 Oct 29 21:53:57.933954 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.933907 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" Oct 29 21:53:57.934008 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.933971 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" event={"ID":"d67f4d1d-4981-4405-8565-e4101c8d1262","Type":"ContainerDied","Data":"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9"} Oct 29 21:53:57.934067 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.934033 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz" event={"ID":"d67f4d1d-4981-4405-8565-e4101c8d1262","Type":"ContainerDied","Data":"df570136311f735df02e69dc3196270a7da8856d1af5f2f96717e3429fac82bc"} Oct 29 21:53:57.941469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.941449 4543 scope.go:122] "RemoveContainer" containerID="f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9" Oct 29 21:53:57.952045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.952028 4543 scope.go:122] "RemoveContainer" containerID="f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9" Oct 29 21:53:57.952352 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:57.952335 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9\": container with ID starting with f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9 not found: ID does not exist" containerID="f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9" Oct 29 21:53:57.952403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.952362 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9"} err="failed to get container status \"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9\": rpc error: code = NotFound desc = could not find container \"f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9\": container with ID starting with f93a4914c3fc82d0e3a576ce67104fee815d31b81a645b22a4a47372c4af7cd9 not found: ID does not exist" Oct 29 21:53:57.962160 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:53:57.962134 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:53:57.990374 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.990351 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:57.996040 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:57.996019 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d56935a-799bc5556c-xmwsz"] Oct 29 21:53:58.031455 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:58.031429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:58.039372 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:58.039354 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d56935a-5557d547bb-6985z"] Oct 29 21:53:58.384087 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:58.384054 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" path="/var/lib/kubelet/pods/4d17ac01-88b7-44fe-bd98-cfa5be9053e4/volumes" Oct 29 21:53:58.384460 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:58.384446 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67f4d1d-4981-4405-8565-e4101c8d1262" path="/var/lib/kubelet/pods/d67f4d1d-4981-4405-8565-e4101c8d1262/volumes" Oct 29 21:53:59.225211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:59.225180 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l"] Oct 29 21:53:59.237188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:53:59.237153 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2976cd0d-sxg7l"] Oct 29 21:54:00.054197 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054168 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 21:54:00.054585 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054572 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" containerName="payload-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054586 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" containerName="payload-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054604 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d67f4d1d-4981-4405-8565-e4101c8d1262" containerName="payload-pre-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054609 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67f4d1d-4981-4405-8565-e4101c8d1262" containerName="payload-pre-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054619 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77539a63-a8d5-4b3b-8940-33638e038942" containerName="payload-pre-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054623 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77539a63-a8d5-4b3b-8940-33638e038942" containerName="payload-pre-processing" Oct 29 21:54:00.054649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054647 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ba146607-c0f2-4b8c-a560-658dce51c606" containerName="payload-processing" Oct 29 21:54:00.054846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054653 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba146607-c0f2-4b8c-a560-658dce51c606" containerName="payload-processing" Oct 29 21:54:00.054846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054726 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67f4d1d-4981-4405-8565-e4101c8d1262" containerName="payload-pre-processing" Oct 29 21:54:00.054846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054734 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77539a63-a8d5-4b3b-8940-33638e038942" containerName="payload-pre-processing" Oct 29 21:54:00.054846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054744 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba146607-c0f2-4b8c-a560-658dce51c606" containerName="payload-processing" Oct 29 21:54:00.054846 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.054752 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d17ac01-88b7-44fe-bd98-cfa5be9053e4" containerName="payload-processing" Oct 29 21:54:00.058275 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.058258 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.077685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.077640 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-92d984-dockercfg-xrnjh\"" Oct 29 21:54:00.089790 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.089769 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-92d984\"" Oct 29 21:54:00.091752 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.091734 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 21:54:00.149705 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.149672 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 21:54:00.156286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.156260 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.160218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.160193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsn9v\" (UniqueName: \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.160515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.160490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.175435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.174987 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 21:54:00.261625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.261592 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsn9v\" (UniqueName: \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.262030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.261661 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.262030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.261688 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhf7\" (UniqueName: \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.262030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.261743 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.262478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.262459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.286405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.286376 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsn9v\" (UniqueName: \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\") pod \"payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.362855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.362797 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.362855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.362827 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhf7\" (UniqueName: \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.363515 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.363492 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.371122 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.371097 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.384368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.384342 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c1af36-3f57-420d-943a-65915e17f337" path="/var/lib/kubelet/pods/b4c1af36-3f57-420d-943a-65915e17f337/volumes" Oct 29 21:54:00.387238 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.387216 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhf7\" (UniqueName: \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\") pod \"payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.491625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.491590 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.679278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.679252 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 21:54:00.681032 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:00.681002 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fc541e_183b_432e_bc68_116f8262171e.slice/crio-2afaf144949869a753b2f20794fed7c7560cea02c9f6878b74ae8e800c2beeb1 WatchSource:0}: Error finding container 2afaf144949869a753b2f20794fed7c7560cea02c9f6878b74ae8e800c2beeb1: Status 404 returned error can't find the container with id 2afaf144949869a753b2f20794fed7c7560cea02c9f6878b74ae8e800c2beeb1 Oct 29 21:54:00.796282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.795835 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 21:54:00.797875 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:00.797847 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162d466f_691b_4409_bd40_fabf0d1f83aa.slice/crio-56c9bcb89f96a8e962162f34d34352363f9c4d2705665bc138ca4ef88508e1d6 WatchSource:0}: Error finding container 56c9bcb89f96a8e962162f34d34352363f9c4d2705665bc138ca4ef88508e1d6: Status 404 returned error can't find the container with id 56c9bcb89f96a8e962162f34d34352363f9c4d2705665bc138ca4ef88508e1d6 Oct 29 21:54:00.889834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.889802 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t"] Oct 29 21:54:00.893555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.893534 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d56935a-mwd9t"] Oct 29 21:54:00.957694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.957598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" event={"ID":"162d466f-691b-4409-bd40-fabf0d1f83aa","Type":"ContainerStarted","Data":"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9"} Oct 29 21:54:00.957694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.957661 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" event={"ID":"162d466f-691b-4409-bd40-fabf0d1f83aa","Type":"ContainerStarted","Data":"56c9bcb89f96a8e962162f34d34352363f9c4d2705665bc138ca4ef88508e1d6"} Oct 29 21:54:00.958390 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.958370 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:00.960403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.960379 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" event={"ID":"29fc541e-183b-432e-bc68-116f8262171e","Type":"ContainerStarted","Data":"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9"} Oct 29 21:54:00.960504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.960422 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:00.960504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:00.960445 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" event={"ID":"29fc541e-183b-432e-bc68-116f8262171e","Type":"ContainerStarted","Data":"2afaf144949869a753b2f20794fed7c7560cea02c9f6878b74ae8e800c2beeb1"} Oct 29 21:54:01.032066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.032015 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" podStartSLOduration=1.032000873 podStartE2EDuration="1.032000873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:01.03132516 +0000 UTC m=+702.773255874" watchObservedRunningTime="2026-10-29 21:54:01.032000873 +0000 UTC m=+702.773931547" Oct 29 21:54:01.091826 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.091774 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" podStartSLOduration=1.091761014 podStartE2EDuration="1.091761014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:01.090112918 +0000 UTC m=+702.832043585" watchObservedRunningTime="2026-10-29 21:54:01.091761014 +0000 UTC m=+702.833691681" Oct 29 21:54:01.539373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.539341 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:54:01.539872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.539567 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" containerID="cri-o://8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d" gracePeriod=30 Oct 29 21:54:01.718882 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.718842 4543 patch_prober.go:29] interesting pod/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 21:54:01.719068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:01.718909 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 21:54:02.383941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.383907 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d11b2a-3df4-468d-a88c-17d848f8b764" path="/var/lib/kubelet/pods/c9d11b2a-3df4-468d-a88c-17d848f8b764/volumes" Oct 29 21:54:02.571035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.571003 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:54:02.571541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.571287 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" containerID="cri-o://9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c" gracePeriod=30 Oct 29 21:54:02.951986 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.951959 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 21:54:02.955435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.955405 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:02.974496 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.974465 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82db63-openshift-default-dockercfg-7p225\"" Oct 29 21:54:02.987822 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:02.987798 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 21:54:03.090260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090297 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090359 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090390 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090436 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8rg\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090544 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090593 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.090691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.090627 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191387 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191354 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191452 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191634 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191700 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191680 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191763 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191744 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8rg\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191861 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191840 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.191861 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191847 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.191954 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.192048 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.192179 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192518 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.192469 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.192518 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.192503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.194058 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.194034 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.194665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.194649 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.215971 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.215951 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.216994 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.216962 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8rg\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.273375 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.273337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:03.594342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.594295 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 21:54:03.597059 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:03.597031 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b837bfb_9a20_436a_b39e_85c1e3daa3e7.slice/crio-1fd02083f1f93c263d3155a18bcc39553de927a7e15a7e517095fa58a51d71b7 WatchSource:0}: Error finding container 1fd02083f1f93c263d3155a18bcc39553de927a7e15a7e517095fa58a51d71b7: Status 404 returned error can't find the container with id 1fd02083f1f93c263d3155a18bcc39553de927a7e15a7e517095fa58a51d71b7 Oct 29 21:54:03.599621 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.599585 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:03.599728 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.599667 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:03.599728 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.599714 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:03.901787 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.901710 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 21:54:03.980928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.980893 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerStarted","Data":"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c"} Oct 29 21:54:03.981152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:03.980960 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerStarted","Data":"1fd02083f1f93c263d3155a18bcc39553de927a7e15a7e517095fa58a51d71b7"} Oct 29 21:54:04.039710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.039661 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podStartSLOduration=2.03964436 podStartE2EDuration="2.03964436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:04.038174153 +0000 UTC m=+705.780104827" watchObservedRunningTime="2026-10-29 21:54:04.03964436 +0000 UTC m=+705.781575033" Oct 29 21:54:04.273512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.273477 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:04.278680 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.278652 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:04.912918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.911634 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 21:54:04.987244 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.987210 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:04.988163 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:04.988140 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:06.555747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.555707 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 21:54:06.559901 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.559874 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.575212 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.575121 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 21:54:06.584355 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.584332 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-863f1b\"" Oct 29 21:54:06.584465 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.584387 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-863f1b-dockercfg-4c2vt\"" Oct 29 21:54:06.627862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.627836 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.627962 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.627899 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92zv\" (UniqueName: \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.653273 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.653250 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 21:54:06.657578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.657518 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:06.662645 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.662606 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 21:54:06.782593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.781049 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.782593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.781181 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92zv\" (UniqueName: \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.782593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.782400 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.806581 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.805812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92zv\" (UniqueName: \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\") pod \"payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.864618 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.864587 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:54:06.881933 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.881901 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.881968 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.881999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882049 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882143 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:06.882295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882181 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882349 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:06.882441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\") pod \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\" (UID: \"a4f2000e-6dbe-4dda-aade-c55688d2e7ca\") " Oct 29 21:54:06.882441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882424 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:54:06.882627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882594 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:06.882681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882660 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:06.882718 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882707 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:06.882775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882758 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdvz\" (UniqueName: \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:06.882937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882922 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.882976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882941 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.882976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882951 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.882976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882960 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.882976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.882968 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.884439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.884417 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "kube-api-access-954zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:06.884764 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.884744 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:06.884897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.884879 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 21:54:06.884956 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.884903 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token" pod "a4f2000e-6dbe-4dda-aade-c55688d2e7ca" (UID: "a4f2000e-6dbe-4dda-aade-c55688d2e7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:06.961936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.961892 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:06.983732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983700 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:06.983835 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983774 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdvz\" (UniqueName: \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:06.983949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983934 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.983982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983956 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954zl\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-kube-api-access-954zl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.983982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983973 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.984054 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.983987 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f2000e-6dbe-4dda-aade-c55688d2e7ca-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:06.984363 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:06.984346 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:07.002311 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.002281 4543 generic.go:356] "Generic (PLEG): container finished" podID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerID="8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d" exitCode=0 Oct 29 21:54:07.002413 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.002337 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" Oct 29 21:54:07.002413 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.002373 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerDied","Data":"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d"} Oct 29 21:54:07.002493 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.002427 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz" event={"ID":"a4f2000e-6dbe-4dda-aade-c55688d2e7ca","Type":"ContainerDied","Data":"b8c13c0bfe056ea0662f5998afa1949f5ee2a3ec25cb26ebfa483f2b2b4c5607"} Oct 29 21:54:07.002493 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.002448 4543 scope.go:122] "RemoveContainer" containerID="8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d" Oct 29 21:54:07.008666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.008639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdvz\" (UniqueName: \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\") pod \"payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:07.012831 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.012812 4543 scope.go:122] "RemoveContainer" containerID="c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd" Oct 29 21:54:07.060347 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.059993 4543 scope.go:122] "RemoveContainer" containerID="8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d" Oct 29 21:54:07.064872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.064815 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:54:07.069119 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:07.068289 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d\": container with ID starting with 8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d not found: ID does not exist" containerID="8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d" Oct 29 21:54:07.069119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.068341 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d"} err="failed to get container status \"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d\": rpc error: code = NotFound desc = could not find container \"8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d\": container with ID starting with 8933dcca2940d65567e91cd10c6919ef50fc96b0c65cf30cacabac01f4208c3d not found: ID does not exist" Oct 29 21:54:07.069119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.068361 4543 scope.go:122] "RemoveContainer" containerID="c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd" Oct 29 21:54:07.069119 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:07.068885 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd\": container with ID starting with c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd not found: ID does not exist" containerID="c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd" Oct 29 21:54:07.069119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.068906 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd"} err="failed to get container status \"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd\": rpc error: code = NotFound desc = could not find container \"c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd\": container with ID starting with c6e6868e8229e849e2b53c7828967d5063a81b9811bde61913fb14924e5188fd not found: ID does not exist" Oct 29 21:54:07.071540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.071485 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f2459168-gw-openshift-default-5995d994f-p4hkz"] Oct 29 21:54:07.276482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.276445 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:07.284639 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.284613 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 21:54:07.288175 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:07.288149 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c09141_c0d9_4e29_a908_c6aedcc10f0e.slice/crio-08ef331980a91250c33373f87e3411825661f5a645951b5fd3b5fa4d6a40671d WatchSource:0}: Error finding container 08ef331980a91250c33373f87e3411825661f5a645951b5fd3b5fa4d6a40671d: Status 404 returned error can't find the container with id 08ef331980a91250c33373f87e3411825661f5a645951b5fd3b5fa4d6a40671d Oct 29 21:54:07.609397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.609373 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 21:54:07.874478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.874455 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:54:07.892848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.892826 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.892952 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.892932 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893026 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893093 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893042 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893096 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893125 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893138 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:07.893298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893158 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893210 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893261 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\") pod \"e81b469b-69ce-4be7-8093-fec68c3365af\" (UID: \"e81b469b-69ce-4be7-8093-fec68c3365af\") " Oct 29 21:54:07.893445 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:07.893445 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893374 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:54:07.893660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893580 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:07.893734 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:07.894159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.893889 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81b469b-69ce-4be7-8093-fec68c3365af-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.894159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.894031 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.894159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.894107 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.894159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.894124 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.894159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.894137 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.895460 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.895424 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 21:54:07.895555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.895460 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "kube-api-access-pvbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:07.895555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.895484 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:07.895555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.895525 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy" pod "e81b469b-69ce-4be7-8093-fec68c3365af" (UID: "e81b469b-69ce-4be7-8093-fec68c3365af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:54:07.994871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.994841 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.994871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.994868 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbrs\" (UniqueName: \"kubernetes.io/projected/e81b469b-69ce-4be7-8093-fec68c3365af-kube-api-access-pvbrs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.994871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.994878 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81b469b-69ce-4be7-8093-fec68c3365af-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:07.995123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:07.994886 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81b469b-69ce-4be7-8093-fec68c3365af-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:54:08.009227 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.009202 4543 generic.go:356] "Generic (PLEG): container finished" podID="e81b469b-69ce-4be7-8093-fec68c3365af" containerID="9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c" exitCode=0 Oct 29 21:54:08.009351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.009277 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerDied","Data":"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c"} Oct 29 21:54:08.009351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.009316 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" event={"ID":"e81b469b-69ce-4be7-8093-fec68c3365af","Type":"ContainerDied","Data":"38ec3265ddb1afebac0c3c37937c3e6373ce2a7cda254b4418777165d1ec4a04"} Oct 29 21:54:08.009351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.009338 4543 scope.go:122] "RemoveContainer" containerID="9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c" Oct 29 21:54:08.009446 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.009409 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw" Oct 29 21:54:08.013844 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.013812 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" event={"ID":"343d315a-d25c-4d90-948b-fbc01f39558f","Type":"ContainerStarted","Data":"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2"} Oct 29 21:54:08.013960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.013853 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" event={"ID":"343d315a-d25c-4d90-948b-fbc01f39558f","Type":"ContainerStarted","Data":"5e7fce859d71df99c5253540154ed3859af974031d38b404fa7525272f3bfa50"} Oct 29 21:54:08.013960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.013874 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:08.015438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.015415 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" event={"ID":"f9c09141-c0d9-4e29-a908-c6aedcc10f0e","Type":"ContainerStarted","Data":"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2"} Oct 29 21:54:08.015511 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.015456 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" event={"ID":"f9c09141-c0d9-4e29-a908-c6aedcc10f0e","Type":"ContainerStarted","Data":"08ef331980a91250c33373f87e3411825661f5a645951b5fd3b5fa4d6a40671d"} Oct 29 21:54:08.015559 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.015517 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:08.025784 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.025765 4543 scope.go:122] "RemoveContainer" containerID="9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3" Oct 29 21:54:08.060116 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.060096 4543 scope.go:122] "RemoveContainer" containerID="9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c" Oct 29 21:54:08.060422 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:08.060393 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c\": container with ID starting with 9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c not found: ID does not exist" containerID="9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c" Oct 29 21:54:08.060478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.060435 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c"} err="failed to get container status \"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c\": rpc error: code = NotFound desc = could not find container \"9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c\": container with ID starting with 9a4b1acacb65d620e349d7adabba9fdabdcd87fcfdebe1e24d364c559505de8c not found: ID does not exist" Oct 29 21:54:08.060478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.060460 4543 scope.go:122] "RemoveContainer" containerID="9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3" Oct 29 21:54:08.060821 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:08.060803 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3\": container with ID starting with 9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3 not found: ID does not exist" containerID="9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3" Oct 29 21:54:08.060877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.060826 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3"} err="failed to get container status \"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3\": rpc error: code = NotFound desc = could not find container \"9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3\": container with ID starting with 9a7f407e8c89c4e411175d5d576efd512cf3fd1bc5d4371c8e5c15f8e98a94e3 not found: ID does not exist" Oct 29 21:54:08.083724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.083673 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" podStartSLOduration=2.083662059 podStartE2EDuration="2.083662059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:08.08324421 +0000 UTC m=+709.825174884" watchObservedRunningTime="2026-10-29 21:54:08.083662059 +0000 UTC m=+709.825592731" Oct 29 21:54:08.122707 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.122656 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:54:08.127903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.127881 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5d56935a-openshift-default-85b4b56bd5-kt4mw"] Oct 29 21:54:08.181326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.181272 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" podStartSLOduration=2.181255986 podStartE2EDuration="2.181255986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:08.180648056 +0000 UTC m=+709.922578730" watchObservedRunningTime="2026-10-29 21:54:08.181255986 +0000 UTC m=+709.923186661" Oct 29 21:54:08.384731 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.384657 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" path="/var/lib/kubelet/pods/a4f2000e-6dbe-4dda-aade-c55688d2e7ca/volumes" Oct 29 21:54:08.385138 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:08.385124 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" path="/var/lib/kubelet/pods/e81b469b-69ce-4be7-8093-fec68c3365af/volumes" Oct 29 21:54:09.366187 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366159 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 21:54:09.366608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366595 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.366655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366609 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.366655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366619 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.366655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366624 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.366655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366637 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.366655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366642 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.366796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366715 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.366796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366727 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.366796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.366733 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f2000e-6dbe-4dda-aade-c55688d2e7ca" containerName="istio-proxy" Oct 29 21:54:09.371212 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.371192 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.389887 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.389844 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 21:54:09.399372 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.399351 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-82db63\"" Oct 29 21:54:09.409817 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.409795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vz8\" (UniqueName: \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.409923 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.409824 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.475188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.472950 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 21:54:09.475188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.473511 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.475188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.473523 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.475188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.473625 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81b469b-69ce-4be7-8093-fec68c3365af" containerName="istio-proxy" Oct 29 21:54:09.477105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.477053 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.478506 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.478486 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-82db63-dockercfg-764xl\"" Oct 29 21:54:09.485793 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.485551 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 21:54:09.511284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.511254 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.511450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.511426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vz8\" (UniqueName: \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.511528 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.511460 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.511528 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.511493 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfld\" (UniqueName: \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.512164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.512145 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.555971 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.555944 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vz8\" (UniqueName: \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\") pod \"payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.612876 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.612848 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfld\" (UniqueName: \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.613011 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.612912 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.613595 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.613571 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.655379 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.655319 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfld\" (UniqueName: \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\") pod \"payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:09.684625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.684596 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:09.801303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:09.801279 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:10.016603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.016572 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 21:54:10.016959 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:10.016934 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a38358d_49a2_4c51_8c99_69929d3d3ef0.slice/crio-6410b0aa24f8fc4b49aa18473bd548ddd592dd15419457fe3298e6377f9624b9 WatchSource:0}: Error finding container 6410b0aa24f8fc4b49aa18473bd548ddd592dd15419457fe3298e6377f9624b9: Status 404 returned error can't find the container with id 6410b0aa24f8fc4b49aa18473bd548ddd592dd15419457fe3298e6377f9624b9 Oct 29 21:54:10.050629 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.050512 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" event={"ID":"2a38358d-49a2-4c51-8c99-69929d3d3ef0","Type":"ContainerStarted","Data":"6410b0aa24f8fc4b49aa18473bd548ddd592dd15419457fe3298e6377f9624b9"} Oct 29 21:54:10.145099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.143450 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 21:54:10.157229 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:10.157180 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe67fc2_a167_4c16_8af1_28d7aee4c728.slice/crio-29f849e62a76df491f34cf9cd2a71e71b1f1d8037d74f199830cbb02ffebfa36 WatchSource:0}: Error finding container 29f849e62a76df491f34cf9cd2a71e71b1f1d8037d74f199830cbb02ffebfa36: Status 404 returned error can't find the container with id 29f849e62a76df491f34cf9cd2a71e71b1f1d8037d74f199830cbb02ffebfa36 Oct 29 21:54:10.394990 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.394919 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 21:54:10.398597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.398577 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.418907 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.418881 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 21:54:10.419876 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.419851 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cea94a-dockercfg-7bh4b\"" Oct 29 21:54:10.420000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.419853 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cea94a\"" Oct 29 21:54:10.421465 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.421433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.421592 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.421525 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p4m9\" (UniqueName: \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.472959 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.472931 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 21:54:10.476898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.476875 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.508590 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.508550 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 21:54:10.523773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.523738 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sqz\" (UniqueName: \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.523944 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.523912 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.524015 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.523958 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.524069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.524009 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4m9\" (UniqueName: \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.525717 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.525690 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.548835 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.548806 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4m9\" (UniqueName: \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\") pod \"payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.625295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.625263 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sqz\" (UniqueName: \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.625472 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.625393 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.627113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.626376 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.650894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.650796 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sqz\" (UniqueName: \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\") pod \"payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:10.712598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.712522 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:10.809114 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:10.809052 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:11.020874 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.020849 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 21:54:11.023694 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:11.023664 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceddcaa9_b308_4a21_bb62_b92bdf980aa9.slice/crio-597cbded6572ef4bde73ecbabe962e7a233fd0182a218ec4bc50a112ae03ba5a WatchSource:0}: Error finding container 597cbded6572ef4bde73ecbabe962e7a233fd0182a218ec4bc50a112ae03ba5a: Status 404 returned error can't find the container with id 597cbded6572ef4bde73ecbabe962e7a233fd0182a218ec4bc50a112ae03ba5a Oct 29 21:54:11.058451 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.058426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" event={"ID":"2a38358d-49a2-4c51-8c99-69929d3d3ef0","Type":"ContainerStarted","Data":"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd"} Oct 29 21:54:11.058561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.058508 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:11.060434 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.060406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" event={"ID":"9fe67fc2-a167-4c16-8af1-28d7aee4c728","Type":"ContainerStarted","Data":"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708"} Oct 29 21:54:11.060525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.060462 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" event={"ID":"9fe67fc2-a167-4c16-8af1-28d7aee4c728","Type":"ContainerStarted","Data":"29f849e62a76df491f34cf9cd2a71e71b1f1d8037d74f199830cbb02ffebfa36"} Oct 29 21:54:11.061047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.061019 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:11.062638 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.062611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" event={"ID":"ceddcaa9-b308-4a21-bb62-b92bdf980aa9","Type":"ContainerStarted","Data":"597cbded6572ef4bde73ecbabe962e7a233fd0182a218ec4bc50a112ae03ba5a"} Oct 29 21:54:11.102488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.102445 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" podStartSLOduration=2.102432168 podStartE2EDuration="2.102432168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:11.10126088 +0000 UTC m=+712.843191557" watchObservedRunningTime="2026-10-29 21:54:11.102432168 +0000 UTC m=+712.844362842" Oct 29 21:54:11.145886 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.145837 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" podStartSLOduration=2.145824251 podStartE2EDuration="2.145824251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:11.143483533 +0000 UTC m=+712.885414206" watchObservedRunningTime="2026-10-29 21:54:11.145824251 +0000 UTC m=+712.887754924" Oct 29 21:54:11.149571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.149547 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 21:54:11.152293 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:11.152266 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a68a24_0c44_4d46_83b4_2469ca64d338.slice/crio-9e4f7cac29c3db8038954b825b4eaa4b11b89401c6c475e8af7ba3bb1ecffaaf WatchSource:0}: Error finding container 9e4f7cac29c3db8038954b825b4eaa4b11b89401c6c475e8af7ba3bb1ecffaaf: Status 404 returned error can't find the container with id 9e4f7cac29c3db8038954b825b4eaa4b11b89401c6c475e8af7ba3bb1ecffaaf Oct 29 21:54:11.971434 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:11.971405 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 21:54:12.071877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.071839 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" event={"ID":"ceddcaa9-b308-4a21-bb62-b92bdf980aa9","Type":"ContainerStarted","Data":"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa"} Oct 29 21:54:12.072028 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.071932 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:12.074047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.074020 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" event={"ID":"b6a68a24-0c44-4d46-83b4-2469ca64d338","Type":"ContainerStarted","Data":"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171"} Oct 29 21:54:12.074176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.074064 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" event={"ID":"b6a68a24-0c44-4d46-83b4-2469ca64d338","Type":"ContainerStarted","Data":"9e4f7cac29c3db8038954b825b4eaa4b11b89401c6c475e8af7ba3bb1ecffaaf"} Oct 29 21:54:12.074286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.074253 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:12.114613 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.114563 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" podStartSLOduration=2.11454702 podStartE2EDuration="2.11454702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:12.113340307 +0000 UTC m=+713.855270981" watchObservedRunningTime="2026-10-29 21:54:12.11454702 +0000 UTC m=+713.856477694" Oct 29 21:54:12.156417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.156368 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" podStartSLOduration=2.156351906 podStartE2EDuration="2.156351906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:12.155219867 +0000 UTC m=+713.897150542" watchObservedRunningTime="2026-10-29 21:54:12.156351906 +0000 UTC m=+713.898282582" Oct 29 21:54:12.976321 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:12.976291 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 21:54:14.088013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.087937 4543 generic.go:356] "Generic (PLEG): container finished" podID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerID="4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600" exitCode=0 Oct 29 21:54:14.088013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.087980 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerDied","Data":"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600"} Oct 29 21:54:14.088491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.088349 4543 scope.go:122] "RemoveContainer" containerID="4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600" Oct 29 21:54:14.089612 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.089578 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:14.089733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.089665 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:14.089733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:14.089726 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:15.096611 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:15.096572 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerStarted","Data":"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc"} Oct 29 21:54:15.307014 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:15.306978 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:54:15.308061 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:15.308030 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 21:54:15.308243 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:15.308116 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 21:54:16.307460 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:16.307421 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 21:54:16.307918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:16.307478 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 21:54:17.307720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:17.307680 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 21:54:17.308297 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:17.307748 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 21:54:17.817064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:17.817029 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:54:18.306840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:18.306807 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 21:54:18.307020 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:18.306868 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 21:54:19.024985 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:19.024940 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 21:54:19.025380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:19.025033 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 21:54:19.307611 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:19.307530 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 29 21:54:19.307611 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:19.307581 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 29 21:54:20.310964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:20.310930 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:54:20.312157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:20.312133 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 21:54:22.077269 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:22.077236 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 21:54:23.082366 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:23.082337 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 21:54:23.082804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:23.082400 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 21:54:23.082804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:23.082429 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 21:54:27.978608 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:27.978570 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:54:33.979965 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:33.979934 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 21:54:33.983853 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:33.983830 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.007225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.007195 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 21:54:34.017155 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.017127 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"simulator-w2-92d984-kserve-self-signed-certs\"" Oct 29 21:54:34.019099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.019048 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"openshift-service-ca.crt\"" Oct 29 21:54:34.019219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.019112 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"default-dockercfg-fkmhr\"" Oct 29 21:54:34.019219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.019142 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"kube-root-ca.crt\"" Oct 29 21:54:34.113497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113459 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.113673 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113612 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.113728 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.113765 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113734 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.113810 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113773 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7tr\" (UniqueName: \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.113810 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.113797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.224748 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.224706 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.225134 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.225107 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.225383 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.225363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.225527 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.225509 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7tr\" (UniqueName: \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.225662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.225645 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.225861 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.225844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.226435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.226414 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.238146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.227448 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.238146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.236949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.238146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.237778 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.238146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.238030 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.251543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.251508 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7tr\" (UniqueName: \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\") pod \"simulator-w2-92d984-kserve-c7956f6c7-htxdf\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.297004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.296957 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:34.611725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:34.611700 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 21:54:34.614844 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:34.614809 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod084ca685_9944_4791_9dbf_e097c7e8cc10.slice/crio-9725f8faae6fe28696f77cda7ee1a8239b7a5d965885a7fd40fb1c858dee69df WatchSource:0}: Error finding container 9725f8faae6fe28696f77cda7ee1a8239b7a5d965885a7fd40fb1c858dee69df: Status 404 returned error can't find the container with id 9725f8faae6fe28696f77cda7ee1a8239b7a5d965885a7fd40fb1c858dee69df Oct 29 21:54:35.227713 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:35.227671 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" event={"ID":"084ca685-9944-4791-9dbf-e097c7e8cc10","Type":"ContainerStarted","Data":"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81"} Oct 29 21:54:35.228168 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:35.227719 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" event={"ID":"084ca685-9944-4791-9dbf-e097c7e8cc10","Type":"ContainerStarted","Data":"9725f8faae6fe28696f77cda7ee1a8239b7a5d965885a7fd40fb1c858dee69df"} Oct 29 21:54:35.270013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:35.269969 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podStartSLOduration=2.2699561409999998 podStartE2EDuration="2.269956141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:35.26917075 +0000 UTC m=+737.011101424" watchObservedRunningTime="2026-10-29 21:54:35.269956141 +0000 UTC m=+737.011886814" Oct 29 21:54:35.297975 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:35.297955 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:36.029218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:36.029186 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:54:36.029393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:36.029293 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:54:37.240462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.240429 4543 generic.go:356] "Generic (PLEG): container finished" podID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerID="864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c" exitCode=0 Oct 29 21:54:37.241001 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.240509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerDied","Data":"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c"} Oct 29 21:54:37.241146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.241130 4543 scope.go:122] "RemoveContainer" containerID="864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c" Oct 29 21:54:37.242289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.242261 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:37.242354 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.242325 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:37.242393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:37.242355 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:38.082809 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.082773 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 21:54:38.085637 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.085619 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.116794 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.116764 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 21:54:38.156436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.156405 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3b067-openshift-default-dockercfg-t4jcs\"" Oct 29 21:54:38.166572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166539 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166611 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166755 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166712 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166791 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166754 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166843 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166868 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.166952 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.166927 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.167038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.167022 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbp2r\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.167157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.167134 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.258001 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.257969 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerStarted","Data":"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016"} Oct 29 21:54:38.268115 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268062 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268149 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268202 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268237 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268261 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbp2r\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268444 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268498 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.268733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.268608 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.269411 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.269052 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.269411 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.269208 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.269411 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.269256 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.269411 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.269346 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.271055 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.271032 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.271420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.271401 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.274494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.274472 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:38.274693 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.274669 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:38.274777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.274719 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:38.293056 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.293035 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbp2r\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.293777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.293756 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.397744 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.397667 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:38.757229 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:38.757199 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c07798b_4181_4248_8f40_78a86dab08d4.slice/crio-c6732f57b545d0365ddc61a3c92ee6b88b2f6fd43d4b61ac9041a5bc1dbf7c77 WatchSource:0}: Error finding container c6732f57b545d0365ddc61a3c92ee6b88b2f6fd43d4b61ac9041a5bc1dbf7c77: Status 404 returned error can't find the container with id c6732f57b545d0365ddc61a3c92ee6b88b2f6fd43d4b61ac9041a5bc1dbf7c77 Oct 29 21:54:38.759198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.759170 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 21:54:38.759649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.759614 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:38.759698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.759681 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:38.759735 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:38.759709 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:54:39.265549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.265509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" event={"ID":"1c07798b-4181-4248-8f40-78a86dab08d4","Type":"ContainerStarted","Data":"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10"} Oct 29 21:54:39.265938 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.265569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" event={"ID":"1c07798b-4181-4248-8f40-78a86dab08d4","Type":"ContainerStarted","Data":"c6732f57b545d0365ddc61a3c92ee6b88b2f6fd43d4b61ac9041a5bc1dbf7c77"} Oct 29 21:54:39.274211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.274182 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:39.274358 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.274235 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:39.313722 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.313679 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podStartSLOduration=1.313666701 podStartE2EDuration="1.313666701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:39.312178828 +0000 UTC m=+741.054109500" watchObservedRunningTime="2026-10-29 21:54:39.313666701 +0000 UTC m=+741.055597433" Oct 29 21:54:39.398222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.398194 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:39.399501 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.399479 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:39.399613 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:39.399529 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:40.280122 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:40.275614 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:40.280122 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:40.275694 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:40.398586 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:40.398556 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:40.398754 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:40.398603 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:41.277583 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:41.277548 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:41.277897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:41.277853 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:41.398775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:41.398741 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:41.399425 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:41.399381 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:42.277519 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:42.277484 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:42.277814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:42.277780 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:42.398949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:42.398919 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:42.399332 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:42.398961 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:43.274726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:43.274687 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 21:54:43.274911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:43.274747 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 21:54:43.398715 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:43.398681 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:43.398879 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:43.398735 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:44.278341 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.278311 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:44.278708 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.278614 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:44.279401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.279380 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 21:54:44.297554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.297527 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:44.309717 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.309697 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:44.327380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.327363 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:54:44.398837 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.398812 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:44.398978 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:44.398870 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:45.398222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:45.398186 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 21:54:45.398600 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:45.398261 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 21:54:46.402957 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:46.402929 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:46.403432 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:46.403187 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:46.403777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:46.403760 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 21:54:56.935143 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.935111 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 21:54:56.938572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.938549 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.958841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.958814 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"openshift-service-ca.crt\"" Oct 29 21:54:56.958972 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.958916 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"kube-root-ca.crt\"" Oct 29 21:54:56.971217 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.971191 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"simulator-w4-cea94a-kserve-self-signed-certs\"" Oct 29 21:54:56.971351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.971309 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"default-dockercfg-s86jj\"" Oct 29 21:54:56.973612 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.973749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973641 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.973749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973686 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.973877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973767 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.973877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973801 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2g2\" (UniqueName: \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.973877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.973836 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:56.975111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:56.975069 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 21:54:57.075009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.074987 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075052 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075116 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2g2\" (UniqueName: \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075144 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075354 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075209 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075476 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075451 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075507 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.075543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.075529 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.077625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.077595 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.077768 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.077754 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.103561 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.103538 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2g2\" (UniqueName: \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\") pod \"simulator-w4-cea94a-kserve-7dbf444464-h46d8\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.250268 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.250236 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:54:57.555824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:57.555738 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 21:54:57.559559 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:54:57.559529 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81e49f31_2aa9_4e89_ac45_a8947ba56599.slice/crio-f7c5284671ee9aecc069ce1b061f89bbc4a8f9ec6121639d64713bbe15465d5b WatchSource:0}: Error finding container f7c5284671ee9aecc069ce1b061f89bbc4a8f9ec6121639d64713bbe15465d5b: Status 404 returned error can't find the container with id f7c5284671ee9aecc069ce1b061f89bbc4a8f9ec6121639d64713bbe15465d5b Oct 29 21:54:57.961182 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:54:57.961148 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:54:58.393599 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:58.393521 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" event={"ID":"81e49f31-2aa9-4e89-ac45-a8947ba56599","Type":"ContainerStarted","Data":"7934379ed9949e09f9aa1f0833354e9e4ad3e1b4421191f11bf7b42e78427830"} Oct 29 21:54:58.393599 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:58.393563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" event={"ID":"81e49f31-2aa9-4e89-ac45-a8947ba56599","Type":"ContainerStarted","Data":"f7c5284671ee9aecc069ce1b061f89bbc4a8f9ec6121639d64713bbe15465d5b"} Oct 29 21:54:58.436680 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:58.436630 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podStartSLOduration=2.436612564 podStartE2EDuration="2.436612564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:58.43592383 +0000 UTC m=+760.177854528" watchObservedRunningTime="2026-10-29 21:54:58.436612564 +0000 UTC m=+760.178543238" Oct 29 21:54:59.250875 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:54:59.250843 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:55:00.033204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.033171 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p"] Oct 29 21:55:00.036121 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.036103 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.057508 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.057486 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p"] Oct 29 21:55:00.066275 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.066236 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:55:00.066275 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.066261 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:55:00.066275 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.066254 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:55:00.066471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.066249 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:55:00.105365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.105344 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.105474 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.105377 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fhb\" (UniqueName: \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.105565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.105546 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.207025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.206990 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.207153 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.207055 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.207153 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.207121 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fhb\" (UniqueName: \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.207692 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.207667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.209382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.209362 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.234193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.234174 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fhb\" (UniqueName: \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\") pod \"maas-api-revoke-sub-1960c30d-hxv2p\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.321552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.321498 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6"] Oct 29 21:55:00.324608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.324593 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.344582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.344561 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6"] Oct 29 21:55:00.349285 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.349267 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:00.408796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.408768 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6sf\" (UniqueName: \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.408931 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.408815 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.409048 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.409011 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.510067 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.510037 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.510216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.510112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6sf\" (UniqueName: \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.510216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.510137 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.510894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.510797 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.512481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.512459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.534368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.534345 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6sf\" (UniqueName: \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\") pod \"maas-api-revoke-sub-bd4b73c5-8tfw6\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.636568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.636511 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:00.653954 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.653935 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p"] Oct 29 21:55:00.656587 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:00.656564 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a18f14_5725_400c_b663_f556bee024f8.slice/crio-44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa WatchSource:0}: Error finding container 44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa: Status 404 returned error can't find the container with id 44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa Oct 29 21:55:00.939414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:00.939390 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6"] Oct 29 21:55:00.942008 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:00.941980 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601d7b0c_7f0a_4c3b_a524_b698fb6ba2aa.slice/crio-0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5 WatchSource:0}: Error finding container 0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5: Status 404 returned error can't find the container with id 0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5 Oct 29 21:55:01.414117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.414070 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" event={"ID":"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa","Type":"ContainerStarted","Data":"544781a9d08c441064d29cc66a2cb40afe505e17830388ec19b6f5e8cb343c92"} Oct 29 21:55:01.414510 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.414148 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" event={"ID":"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa","Type":"ContainerStarted","Data":"0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5"} Oct 29 21:55:01.415712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.415685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" event={"ID":"e6a18f14-5725-400c-b663-f556bee024f8","Type":"ContainerStarted","Data":"6b0974f4c46553028d0c7137db40a3985410bd42991a02018a81709b7211f10e"} Oct 29 21:55:01.415834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.415731 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" event={"ID":"e6a18f14-5725-400c-b663-f556bee024f8","Type":"ContainerStarted","Data":"44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa"} Oct 29 21:55:01.456553 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.456512 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" podStartSLOduration=1.456495311 podStartE2EDuration="1.456495311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:01.45444411 +0000 UTC m=+763.196374784" watchObservedRunningTime="2026-10-29 21:55:01.456495311 +0000 UTC m=+763.198425984" Oct 29 21:55:01.496974 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:01.496897 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" podStartSLOduration=1.496885655 podStartE2EDuration="1.496885655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:01.496198303 +0000 UTC m=+763.238128976" watchObservedRunningTime="2026-10-29 21:55:01.496885655 +0000 UTC m=+763.238816328" Oct 29 21:55:02.421969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:02.421935 4543 generic.go:356] "Generic (PLEG): container finished" podID="601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" containerID="544781a9d08c441064d29cc66a2cb40afe505e17830388ec19b6f5e8cb343c92" exitCode=0 Oct 29 21:55:02.422417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:02.422025 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" event={"ID":"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa","Type":"ContainerDied","Data":"544781a9d08c441064d29cc66a2cb40afe505e17830388ec19b6f5e8cb343c92"} Oct 29 21:55:02.423591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:02.423570 4543 generic.go:356] "Generic (PLEG): container finished" podID="e6a18f14-5725-400c-b663-f556bee024f8" containerID="6b0974f4c46553028d0c7137db40a3985410bd42991a02018a81709b7211f10e" exitCode=0 Oct 29 21:55:02.423703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:02.423627 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" event={"ID":"e6a18f14-5725-400c-b663-f556bee024f8","Type":"ContainerDied","Data":"6b0974f4c46553028d0c7137db40a3985410bd42991a02018a81709b7211f10e"} Oct 29 21:55:03.593544 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.593523 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:03.597001 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.596978 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:03.643572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643550 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\") pod \"e6a18f14-5725-400c-b663-f556bee024f8\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " Oct 29 21:55:03.643684 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643588 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\") pod \"e6a18f14-5725-400c-b663-f556bee024f8\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " Oct 29 21:55:03.643684 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643608 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\") pod \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " Oct 29 21:55:03.643812 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643724 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\") pod \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " Oct 29 21:55:03.643879 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\" (UniqueName: \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\") pod \"e6a18f14-5725-400c-b663-f556bee024f8\" (UID: \"e6a18f14-5725-400c-b663-f556bee024f8\") " Oct 29 21:55:03.643966 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643947 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca" pod "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" (UID: "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:03.644024 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643958 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca" pod "e6a18f14-5725-400c-b663-f556bee024f8" (UID: "e6a18f14-5725-400c-b663-f556bee024f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:03.644024 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.643948 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\" (UniqueName: \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\") pod \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\" (UID: \"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa\") " Oct 29 21:55:03.644462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.644443 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:03.644540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.644467 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:03.646437 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.646410 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca" pod "e6a18f14-5725-400c-b663-f556bee024f8" (UID: "e6a18f14-5725-400c-b663-f556bee024f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:03.646685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.646665 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf" pod "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" (UID: "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa"). InnerVolumeSpecName "kube-api-access-4n6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:03.646774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.646743 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb" pod "e6a18f14-5725-400c-b663-f556bee024f8" (UID: "e6a18f14-5725-400c-b663-f556bee024f8"). InnerVolumeSpecName "kube-api-access-l8fhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:03.647177 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.647155 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca" pod "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" (UID: "601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:03.745158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.745133 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6sf\" (UniqueName: \"kubernetes.io/projected/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-kube-api-access-4n6sf\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:03.745158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.745156 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a18f14-5725-400c-b663-f556bee024f8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:03.745300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.745165 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:03.745300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:03.745173 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fhb\" (UniqueName: \"kubernetes.io/projected/e6a18f14-5725-400c-b663-f556bee024f8-kube-api-access-l8fhb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:04.436710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.436677 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" Oct 29 21:55:04.436710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.436698 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6" event={"ID":"601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa","Type":"ContainerDied","Data":"0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5"} Oct 29 21:55:04.436929 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.436748 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6627706a440907f7a5469d2e6b5cfcea8b89fb2bb0fab82bf168555b028dc5" Oct 29 21:55:04.438222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.438204 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" Oct 29 21:55:04.438352 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.438244 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p" event={"ID":"e6a18f14-5725-400c-b663-f556bee024f8","Type":"ContainerDied","Data":"44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa"} Oct 29 21:55:04.438352 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:04.438285 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44bf6251b4be1ca0a7d2a5ca9edf25cfbf2e00d22a3711305484db0f9bca1daa" Oct 29 21:55:07.250986 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:07.250954 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:55:07.271518 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:07.271490 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:55:07.478329 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:07.478030 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:55:08.863773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.863734 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 21:55:08.864530 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864507 4543 cpu_manager.go:414] "containerMap: removing container" podUID="601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" containerName="revoke-keys" Oct 29 21:55:08.864591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864535 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" containerName="revoke-keys" Oct 29 21:55:08.864591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864576 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e6a18f14-5725-400c-b663-f556bee024f8" containerName="revoke-keys" Oct 29 21:55:08.864591 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864585 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a18f14-5725-400c-b663-f556bee024f8" containerName="revoke-keys" Oct 29 21:55:08.864754 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864740 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" containerName="revoke-keys" Oct 29 21:55:08.864808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.864765 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a18f14-5725-400c-b663-f556bee024f8" containerName="revoke-keys" Oct 29 21:55:08.867927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.867904 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:08.897292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.897270 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"premium-w2-92d984-kserve-self-signed-certs\"" Oct 29 21:55:08.899578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:08.899557 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 21:55:09.003252 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003226 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.003367 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003269 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.003429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003395 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.003489 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.003534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003506 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.003615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.003547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdpk\" (UniqueName: \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104665 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104717 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdpk\" (UniqueName: \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104833 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.104889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.104882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.105214 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.105144 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.105214 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.105198 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.105300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.105263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.107170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.107148 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.107520 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.107505 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.129452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.129401 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdpk\" (UniqueName: \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\") pod \"premium-w2-92d984-kserve-5c54d8b696-9868r\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.182521 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.182499 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:09.490514 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:09.490491 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 21:55:09.493207 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:09.493179 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987893f5_5826_4d00_8371_826c402525cb.slice/crio-db5baab582b4fc14d5a5dca2360a2bc0a9532ab85871b5d56422649df77a10ca WatchSource:0}: Error finding container db5baab582b4fc14d5a5dca2360a2bc0a9532ab85871b5d56422649df77a10ca: Status 404 returned error can't find the container with id db5baab582b4fc14d5a5dca2360a2bc0a9532ab85871b5d56422649df77a10ca Oct 29 21:55:10.480502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:10.480467 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" event={"ID":"987893f5-5826-4d00-8371-826c402525cb","Type":"ContainerStarted","Data":"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52"} Oct 29 21:55:10.480892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:10.480510 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" event={"ID":"987893f5-5826-4d00-8371-826c402525cb","Type":"ContainerStarted","Data":"db5baab582b4fc14d5a5dca2360a2bc0a9532ab85871b5d56422649df77a10ca"} Oct 29 21:55:10.523636 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:10.523587 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podStartSLOduration=2.523573041 podStartE2EDuration="2.523573041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:10.522947836 +0000 UTC m=+772.264878510" watchObservedRunningTime="2026-10-29 21:55:10.523573041 +0000 UTC m=+772.265503717" Oct 29 21:55:11.183283 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:11.183200 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:14.907558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.907519 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g"] Oct 29 21:55:14.912146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.912125 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:14.931423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.931386 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g"] Oct 29 21:55:14.942721 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.940255 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:55:14.942721 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.940376 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:55:14.942721 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.940442 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:55:14.942721 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:14.940566 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:55:15.060050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.060020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dq2\" (UniqueName: \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.060181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.060154 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.060233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.060201 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.094930 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.094906 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7"] Oct 29 21:55:15.098651 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.098633 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.117608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.117582 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7"] Oct 29 21:55:15.161748 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.161650 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dq2\" (UniqueName: \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.161748 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.161729 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.161991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.161826 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.161991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.161919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.161991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.161972 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm58j\" (UniqueName: \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.162196 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.162036 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.163179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.163152 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.165608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.165578 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.186626 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.186604 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dq2\" (UniqueName: \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\") pod \"maas-api-revoke-sub-d36b68b7-pt87g\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.226478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.226454 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:15.262727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.262695 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm58j\" (UniqueName: \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.262840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.262749 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.262840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.262816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.263417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.263393 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.265152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.265134 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.291762 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.291703 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm58j\" (UniqueName: \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\") pod \"maas-api-revoke-sub-ee90ee1c-94kr7\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.411488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.411456 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:15.532109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.532058 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g"] Oct 29 21:55:15.533827 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:15.533794 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb624a598_0a3b_499c_9836_d9a0b8cb1b89.slice/crio-4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c WatchSource:0}: Error finding container 4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c: Status 404 returned error can't find the container with id 4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c Oct 29 21:55:15.714212 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:15.714192 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7"] Oct 29 21:55:15.716277 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:15.716234 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71faa10d_4060_46a6_8b9d_01180642ad24.slice/crio-644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2 WatchSource:0}: Error finding container 644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2: Status 404 returned error can't find the container with id 644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2 Oct 29 21:55:16.521021 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.520911 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" event={"ID":"71faa10d-4060-46a6-8b9d-01180642ad24","Type":"ContainerStarted","Data":"e3bc1e71c448ae7a289b2a359f78880fd24781c4b487e7d1bc8669c56e3549ee"} Oct 29 21:55:16.521021 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.520988 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" event={"ID":"71faa10d-4060-46a6-8b9d-01180642ad24","Type":"ContainerStarted","Data":"644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2"} Oct 29 21:55:16.522795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.522767 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" event={"ID":"b624a598-0a3b-499c-9836-d9a0b8cb1b89","Type":"ContainerStarted","Data":"2be410dd74119a14ed39cd832dfa78b12356b2aeb4da4507afc3eda9ece26f0f"} Oct 29 21:55:16.522910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.522817 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" event={"ID":"b624a598-0a3b-499c-9836-d9a0b8cb1b89","Type":"ContainerStarted","Data":"4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c"} Oct 29 21:55:16.563558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.563512 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" podStartSLOduration=1.563498535 podStartE2EDuration="1.563498535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:16.562260171 +0000 UTC m=+778.304190844" watchObservedRunningTime="2026-10-29 21:55:16.563498535 +0000 UTC m=+778.305429207" Oct 29 21:55:16.603462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:16.603413 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" podStartSLOduration=2.603400229 podStartE2EDuration="2.603400229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:16.602948626 +0000 UTC m=+778.344879299" watchObservedRunningTime="2026-10-29 21:55:16.603400229 +0000 UTC m=+778.345330902" Oct 29 21:55:17.531200 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:17.531163 4543 generic.go:356] "Generic (PLEG): container finished" podID="b624a598-0a3b-499c-9836-d9a0b8cb1b89" containerID="2be410dd74119a14ed39cd832dfa78b12356b2aeb4da4507afc3eda9ece26f0f" exitCode=0 Oct 29 21:55:17.531640 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:17.531246 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" event={"ID":"b624a598-0a3b-499c-9836-d9a0b8cb1b89","Type":"ContainerDied","Data":"2be410dd74119a14ed39cd832dfa78b12356b2aeb4da4507afc3eda9ece26f0f"} Oct 29 21:55:17.532970 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:17.532937 4543 generic.go:356] "Generic (PLEG): container finished" podID="71faa10d-4060-46a6-8b9d-01180642ad24" containerID="e3bc1e71c448ae7a289b2a359f78880fd24781c4b487e7d1bc8669c56e3549ee" exitCode=0 Oct 29 21:55:17.533141 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:17.532989 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" event={"ID":"71faa10d-4060-46a6-8b9d-01180642ad24","Type":"ContainerDied","Data":"e3bc1e71c448ae7a289b2a359f78880fd24781c4b487e7d1bc8669c56e3549ee"} Oct 29 21:55:18.680740 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.680713 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:18.705008 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.704988 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:18.800758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.800692 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\") pod \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " Oct 29 21:55:18.800885 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.800802 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\" (UniqueName: \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\") pod \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " Oct 29 21:55:18.800982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.800964 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\") pod \"71faa10d-4060-46a6-8b9d-01180642ad24\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " Oct 29 21:55:18.801045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.801030 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\") pod \"71faa10d-4060-46a6-8b9d-01180642ad24\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " Oct 29 21:55:18.801110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.801066 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\" (UniqueName: \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\") pod \"71faa10d-4060-46a6-8b9d-01180642ad24\" (UID: \"71faa10d-4060-46a6-8b9d-01180642ad24\") " Oct 29 21:55:18.801190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.801171 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\") pod \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\" (UID: \"b624a598-0a3b-499c-9836-d9a0b8cb1b89\") " Oct 29 21:55:18.801889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.801616 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca" pod "71faa10d-4060-46a6-8b9d-01180642ad24" (UID: "71faa10d-4060-46a6-8b9d-01180642ad24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:18.802069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.801957 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca" pod "b624a598-0a3b-499c-9836-d9a0b8cb1b89" (UID: "b624a598-0a3b-499c-9836-d9a0b8cb1b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:18.802069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.802016 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:18.803227 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.803204 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca" pod "b624a598-0a3b-499c-9836-d9a0b8cb1b89" (UID: "b624a598-0a3b-499c-9836-d9a0b8cb1b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:18.803408 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.803383 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2" pod "b624a598-0a3b-499c-9836-d9a0b8cb1b89" (UID: "b624a598-0a3b-499c-9836-d9a0b8cb1b89"). InnerVolumeSpecName "kube-api-access-s7dq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:18.803771 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.803753 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca" pod "71faa10d-4060-46a6-8b9d-01180642ad24" (UID: "71faa10d-4060-46a6-8b9d-01180642ad24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:18.803970 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.803956 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j" pod "71faa10d-4060-46a6-8b9d-01180642ad24" (UID: "71faa10d-4060-46a6-8b9d-01180642ad24"). InnerVolumeSpecName "kube-api-access-mm58j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:18.903302 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.903278 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71faa10d-4060-46a6-8b9d-01180642ad24-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:18.903391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.903304 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm58j\" (UniqueName: \"kubernetes.io/projected/71faa10d-4060-46a6-8b9d-01180642ad24-kube-api-access-mm58j\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:18.903391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.903314 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:18.903391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.903322 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b624a598-0a3b-499c-9836-d9a0b8cb1b89-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:18.903391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:18.903330 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7dq2\" (UniqueName: \"kubernetes.io/projected/b624a598-0a3b-499c-9836-d9a0b8cb1b89-kube-api-access-s7dq2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:19.183525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.183467 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:19.195779 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.195756 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:19.544202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.544175 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" Oct 29 21:55:19.544388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.544213 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g" event={"ID":"b624a598-0a3b-499c-9836-d9a0b8cb1b89","Type":"ContainerDied","Data":"4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c"} Oct 29 21:55:19.544388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.544256 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4acf4f88f096a6763d5abb4cbcde805f6eb3be8a53b1d33ca2af29294504517c" Oct 29 21:55:19.545793 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.545774 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" Oct 29 21:55:19.545936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.545806 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7" event={"ID":"71faa10d-4060-46a6-8b9d-01180642ad24","Type":"ContainerDied","Data":"644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2"} Oct 29 21:55:19.545936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.545852 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644c0d3ac5d6b1531d7b46d0b7b97742761185fc4eab420f097b5d70141cbdf2" Oct 29 21:55:19.559442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:19.559425 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:55:24.279175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.279136 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 21:55:24.279834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.279812 4543 cpu_manager.go:414] "containerMap: removing container" podUID="71faa10d-4060-46a6-8b9d-01180642ad24" containerName="revoke-keys" Oct 29 21:55:24.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.279834 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71faa10d-4060-46a6-8b9d-01180642ad24" containerName="revoke-keys" Oct 29 21:55:24.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.279863 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b624a598-0a3b-499c-9836-d9a0b8cb1b89" containerName="revoke-keys" Oct 29 21:55:24.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.279872 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b624a598-0a3b-499c-9836-d9a0b8cb1b89" containerName="revoke-keys" Oct 29 21:55:24.280126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.280031 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b624a598-0a3b-499c-9836-d9a0b8cb1b89" containerName="revoke-keys" Oct 29 21:55:24.280126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.280055 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="71faa10d-4060-46a6-8b9d-01180642ad24" containerName="revoke-keys" Oct 29 21:55:24.283642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.283620 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.314166 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.314145 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-92d984-openshift-default-dockercfg-5jl7t\"" Oct 29 21:55:24.317167 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.317148 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 21:55:24.453547 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4r7f\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453595 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453650 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453757 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453912 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453853 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.453977 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.453954 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.554941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.554867 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.554941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.554919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.554944 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.554970 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555020 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555063 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4r7f\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555327 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555389 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555503 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555151 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555511 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555557 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555709 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555670 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.555811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.555790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.557683 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.557662 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.557867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.557851 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.591451 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.591424 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.591716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.591694 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4r7f\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.597345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.597329 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:24.916447 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.916428 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 21:55:24.918141 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:24.918114 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d6ee3d_b0d6_4b9f_8c54_f2b2cb307701.slice/crio-bbf6a0900f0eb2666d0dfe47f1742115328ad7f108125691ef3b78216024b9d8 WatchSource:0}: Error finding container bbf6a0900f0eb2666d0dfe47f1742115328ad7f108125691ef3b78216024b9d8: Status 404 returned error can't find the container with id bbf6a0900f0eb2666d0dfe47f1742115328ad7f108125691ef3b78216024b9d8 Oct 29 21:55:24.920307 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.920274 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:55:24.920373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.920338 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:55:24.920373 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:24.920367 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:55:25.602596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:25.602563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" event={"ID":"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701","Type":"ContainerStarted","Data":"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d"} Oct 29 21:55:25.602596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:25.602613 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" event={"ID":"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701","Type":"ContainerStarted","Data":"bbf6a0900f0eb2666d0dfe47f1742115328ad7f108125691ef3b78216024b9d8"} Oct 29 21:55:25.675250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:25.675200 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podStartSLOduration=1.675184921 podStartE2EDuration="1.675184921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:25.672413962 +0000 UTC m=+787.414344638" watchObservedRunningTime="2026-10-29 21:55:25.675184921 +0000 UTC m=+787.417115599" Oct 29 21:55:26.598401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:26.598365 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:26.599556 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:26.599527 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 21:55:26.599664 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:26.599575 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 21:55:27.598034 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:27.597994 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 21:55:27.598436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:27.598057 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 21:55:27.989498 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:55:27.989465 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:55:28.597999 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:28.597966 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 21:55:28.598289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:28.598024 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 21:55:29.598424 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:29.598382 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 21:55:29.598807 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:29.598442 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 21:55:30.597991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:30.597960 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 29 21:55:30.598184 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:30.598010 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 29 21:55:31.821303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:31.821274 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:31.821738 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:31.821551 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:31.822330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:31.822315 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 21:55:32.996186 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:32.996159 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs"] Oct 29 21:55:32.999803 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:32.999785 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.020450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.020430 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs"] Oct 29 21:55:33.030325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.030301 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:55:33.030325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.030319 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:55:33.030325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.030304 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:55:33.030496 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.030313 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:55:33.156387 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.156366 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.156496 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.156427 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.156578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.156557 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw9m\" (UniqueName: \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.258219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.258130 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.258361 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.258229 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw9m\" (UniqueName: \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.258361 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.258283 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.258918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.258899 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.260677 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.260658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.284009 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.283987 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw9m\" (UniqueName: \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\") pod \"maas-api-revoke-sub-cd5e7470-fx5cs\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.313136 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.313114 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:33.617704 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.617681 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs"] Oct 29 21:55:33.620046 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:33.620015 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86c0885_9d66_4daf_ac04_327ad505cf57.slice/crio-8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6 WatchSource:0}: Error finding container 8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6: Status 404 returned error can't find the container with id 8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6 Oct 29 21:55:33.678949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:33.678921 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" event={"ID":"d86c0885-9d66-4daf-ac04-327ad505cf57","Type":"ContainerStarted","Data":"8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6"} Oct 29 21:55:34.686053 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:34.686016 4543 generic.go:356] "Generic (PLEG): container finished" podID="d86c0885-9d66-4daf-ac04-327ad505cf57" containerID="cbfdd74cf75f382cb5114046f902376c6fae21ae7198576a3ae0b0f3425529e1" exitCode=0 Oct 29 21:55:34.686387 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:34.686105 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" event={"ID":"d86c0885-9d66-4daf-ac04-327ad505cf57","Type":"ContainerDied","Data":"cbfdd74cf75f382cb5114046f902376c6fae21ae7198576a3ae0b0f3425529e1"} Oct 29 21:55:35.845357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.845334 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:35.983792 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.983755 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\") pod \"d86c0885-9d66-4daf-ac04-327ad505cf57\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " Oct 29 21:55:35.983949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.983868 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\" (UniqueName: \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\") pod \"d86c0885-9d66-4daf-ac04-327ad505cf57\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " Oct 29 21:55:35.983949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.983904 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\") pod \"d86c0885-9d66-4daf-ac04-327ad505cf57\" (UID: \"d86c0885-9d66-4daf-ac04-327ad505cf57\") " Oct 29 21:55:35.984187 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.984163 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca" pod "d86c0885-9d66-4daf-ac04-327ad505cf57" (UID: "d86c0885-9d66-4daf-ac04-327ad505cf57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:35.986028 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.986001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m" pod "d86c0885-9d66-4daf-ac04-327ad505cf57" (UID: "d86c0885-9d66-4daf-ac04-327ad505cf57"). InnerVolumeSpecName "kube-api-access-lqw9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:35.986265 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:35.986245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca" pod "d86c0885-9d66-4daf-ac04-327ad505cf57" (UID: "d86c0885-9d66-4daf-ac04-327ad505cf57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:36.085355 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.085334 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:36.085463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.085357 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c0885-9d66-4daf-ac04-327ad505cf57-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:36.085463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.085368 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw9m\" (UniqueName: \"kubernetes.io/projected/d86c0885-9d66-4daf-ac04-327ad505cf57-kube-api-access-lqw9m\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:36.698558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.698491 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" Oct 29 21:55:36.698558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.698522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs" event={"ID":"d86c0885-9d66-4daf-ac04-327ad505cf57","Type":"ContainerDied","Data":"8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6"} Oct 29 21:55:36.698720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:36.698573 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8289e15f0e07a64beb1c8e61241f66887a04caab09d9387eece56c1f1d1185d6" Oct 29 21:55:43.354172 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.354139 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll"] Oct 29 21:55:43.354673 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.354658 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d86c0885-9d66-4daf-ac04-327ad505cf57" containerName="revoke-keys" Oct 29 21:55:43.354720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.354674 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86c0885-9d66-4daf-ac04-327ad505cf57" containerName="revoke-keys" Oct 29 21:55:43.354791 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.354781 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86c0885-9d66-4daf-ac04-327ad505cf57" containerName="revoke-keys" Oct 29 21:55:43.358175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.358158 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.377297 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.377275 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll"] Oct 29 21:55:43.387927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.387903 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:55:43.387927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.387918 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:55:43.388170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.387949 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:55:43.388170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.387956 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:55:43.457449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.457429 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.457545 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.457476 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.457618 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.457600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvzs\" (UniqueName: \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.558121 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.558069 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvzs\" (UniqueName: \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.558224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.558184 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.558224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.558220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.558876 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.558840 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.560745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.560720 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.584218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.584199 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvzs\" (UniqueName: \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\") pod \"maas-api-revoke-sub-86b493e1-92tll\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.670417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.670364 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:43.973630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:43.973562 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll"] Oct 29 21:55:43.974143 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:43.974119 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b99714e_680c_4eac_905d_5cfa16915d86.slice/crio-0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e WatchSource:0}: Error finding container 0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e: Status 404 returned error can't find the container with id 0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e Oct 29 21:55:44.753955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:44.753921 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" event={"ID":"8b99714e-680c-4eac-905d-5cfa16915d86","Type":"ContainerStarted","Data":"29a98069ff447468ede39a6c2f14baff7110d19a5409af8002a42ad040eacdf2"} Oct 29 21:55:44.754364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:44.753986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" event={"ID":"8b99714e-680c-4eac-905d-5cfa16915d86","Type":"ContainerStarted","Data":"0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e"} Oct 29 21:55:44.814468 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:44.814425 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" podStartSLOduration=1.814411902 podStartE2EDuration="1.814411902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:44.81375531 +0000 UTC m=+806.555685986" watchObservedRunningTime="2026-10-29 21:55:44.814411902 +0000 UTC m=+806.556342576" Oct 29 21:55:45.761181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:45.761143 4543 generic.go:356] "Generic (PLEG): container finished" podID="8b99714e-680c-4eac-905d-5cfa16915d86" containerID="29a98069ff447468ede39a6c2f14baff7110d19a5409af8002a42ad040eacdf2" exitCode=0 Oct 29 21:55:45.761545 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:45.761211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" event={"ID":"8b99714e-680c-4eac-905d-5cfa16915d86","Type":"ContainerDied","Data":"29a98069ff447468ede39a6c2f14baff7110d19a5409af8002a42ad040eacdf2"} Oct 29 21:55:46.916710 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.916632 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:46.991151 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.991117 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\" (UniqueName: \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\") pod \"8b99714e-680c-4eac-905d-5cfa16915d86\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " Oct 29 21:55:46.991289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.991173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\") pod \"8b99714e-680c-4eac-905d-5cfa16915d86\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " Oct 29 21:55:46.991289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.991241 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\") pod \"8b99714e-680c-4eac-905d-5cfa16915d86\" (UID: \"8b99714e-680c-4eac-905d-5cfa16915d86\") " Oct 29 21:55:46.991618 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.991591 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca" pod "8b99714e-680c-4eac-905d-5cfa16915d86" (UID: "8b99714e-680c-4eac-905d-5cfa16915d86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:55:46.991819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.991767 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:46.996830 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.996806 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs" pod "8b99714e-680c-4eac-905d-5cfa16915d86" (UID: "8b99714e-680c-4eac-905d-5cfa16915d86"). InnerVolumeSpecName "kube-api-access-ncvzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:55:46.997001 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:46.996981 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca" pod "8b99714e-680c-4eac-905d-5cfa16915d86" (UID: "8b99714e-680c-4eac-905d-5cfa16915d86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:55:47.092550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:47.092489 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvzs\" (UniqueName: \"kubernetes.io/projected/8b99714e-680c-4eac-905d-5cfa16915d86-kube-api-access-ncvzs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:47.092550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:47.092515 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99714e-680c-4eac-905d-5cfa16915d86-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:55:47.774331 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:47.774285 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" event={"ID":"8b99714e-680c-4eac-905d-5cfa16915d86","Type":"ContainerDied","Data":"0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e"} Oct 29 21:55:47.774331 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:47.774326 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll" Oct 29 21:55:47.774540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:47.774340 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0905b5c7df89c7b3ab4d01ce568c0afe9c8f06e339c0deafd28ba90ecccfb23e" Oct 29 21:55:57.959922 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:55:57.959883 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:55:58.098004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.097970 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:55:58.098759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.098738 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8b99714e-680c-4eac-905d-5cfa16915d86" containerName="revoke-keys" Oct 29 21:55:58.098759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.098758 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99714e-680c-4eac-905d-5cfa16915d86" containerName="revoke-keys" Oct 29 21:55:58.098935 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.098920 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99714e-680c-4eac-905d-5cfa16915d86" containerName="revoke-keys" Oct 29 21:55:58.102654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.102588 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.120197 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.120175 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:55:58.131292 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.131262 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"default-dockercfg-g5d5n\"" Oct 29 21:55:58.131393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.131308 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"kube-root-ca.crt\"" Oct 29 21:55:58.133184 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.133169 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"e2e-conflict-a-fb842e5c-kserve-self-signed-certs\"" Oct 29 21:55:58.133237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.133179 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"openshift-service-ca.crt\"" Oct 29 21:55:58.200999 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.200973 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.201116 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.201013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t592q\" (UniqueName: \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.201116 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.201037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.201196 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.201177 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.201271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.201258 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.201335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.201318 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.302200 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.302162 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.302439 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.302220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t592q\" (UniqueName: \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.302740 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.302807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.303014 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.303098 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303520 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.303338 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303663 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.303638 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.303744 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.303661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.306994 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.306105 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.319812 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.319785 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.330281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.330253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t592q\" (UniqueName: \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\") pod \"e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.416186 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.416154 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:55:58.726124 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.726104 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:55:58.727821 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:55:58.727795 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850980f8_dd9e_4f43_95ea_bea0cab17f03.slice/crio-e3f42fb43eeb3d8bc9d42e02b946112a1e28cb3846f1ca6b8f848ece03617baa WatchSource:0}: Error finding container e3f42fb43eeb3d8bc9d42e02b946112a1e28cb3846f1ca6b8f848ece03617baa: Status 404 returned error can't find the container with id e3f42fb43eeb3d8bc9d42e02b946112a1e28cb3846f1ca6b8f848ece03617baa Oct 29 21:55:58.849615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.849577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" event={"ID":"850980f8-dd9e-4f43-95ea-bea0cab17f03","Type":"ContainerStarted","Data":"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1"} Oct 29 21:55:58.849774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.849631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" event={"ID":"850980f8-dd9e-4f43-95ea-bea0cab17f03","Type":"ContainerStarted","Data":"e3f42fb43eeb3d8bc9d42e02b946112a1e28cb3846f1ca6b8f848ece03617baa"} Oct 29 21:55:58.895622 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:58.895576 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" podStartSLOduration=0.895562377 podStartE2EDuration="895.562377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:58.895180061 +0000 UTC m=+820.637110736" watchObservedRunningTime="2026-10-29 21:55:58.895562377 +0000 UTC m=+820.637493049" Oct 29 21:55:59.416455 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:55:59.416426 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:56:08.416568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:08.416536 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:56:08.429444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:08.429417 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:56:08.917033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:08.917004 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:56:11.075635 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.075564 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt"] Oct 29 21:56:11.079682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.079662 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.100048 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.100027 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt"] Oct 29 21:56:11.109595 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.109570 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:56:11.109595 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.109586 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:56:11.110230 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.110214 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:56:11.110331 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.110317 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:56:11.228359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.228335 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.228486 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.228464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54c2w\" (UniqueName: \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.228538 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.228504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.329842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.329786 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.329931 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.329905 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54c2w\" (UniqueName: \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.329976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.329944 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.330615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.330554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.332296 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.332277 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.355641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.355619 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54c2w\" (UniqueName: \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\") pod \"maas-api-revoke-sub-06b1a519-p5xkt\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.394414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.394393 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:11.696597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.696574 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt"] Oct 29 21:56:11.698675 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:56:11.698638 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf364c227_13d2_4b69_92a1_854614113206.slice/crio-6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6 WatchSource:0}: Error finding container 6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6: Status 404 returned error can't find the container with id 6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6 Oct 29 21:56:11.891599 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.891441 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns"] Oct 29 21:56:11.896492 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.896474 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:11.916328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.916299 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns"] Oct 29 21:56:11.934170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:11.934132 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" event={"ID":"f364c227-13d2-4b69-92a1-854614113206","Type":"ContainerStarted","Data":"6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6"} Oct 29 21:56:12.037404 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.037376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppgh\" (UniqueName: \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.037500 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.037417 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.037500 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.037494 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.138946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.138911 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppgh\" (UniqueName: \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.138946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.138947 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.139380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.138993 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.139823 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.139800 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.141521 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.141500 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.164798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.164776 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppgh\" (UniqueName: \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\") pod \"maas-api-revoke-sub-bfb3c40f-7f6ns\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.209642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.209622 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:12.514177 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.514149 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns"] Oct 29 21:56:12.940969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.940933 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" event={"ID":"8f3bc952-bbf9-4c61-b83f-cf4d65acf471","Type":"ContainerStarted","Data":"8e69bc6a48e0e216decc518c00769b7d0f97053c82ef4e2a93f3231bc894ce79"} Oct 29 21:56:12.941136 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.940996 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" event={"ID":"8f3bc952-bbf9-4c61-b83f-cf4d65acf471","Type":"ContainerStarted","Data":"8ff61579227b8d28206f2acfa26a05322de5cb95b686bfaa4defe880885689b8"} Oct 29 21:56:12.942594 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.942568 4543 generic.go:356] "Generic (PLEG): container finished" podID="f364c227-13d2-4b69-92a1-854614113206" containerID="acb2519768f4a8cbd95d9eaf2ca7cf1d6c780654d921d2e93a41a8ddb3c42983" exitCode=0 Oct 29 21:56:12.942688 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.942640 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" event={"ID":"f364c227-13d2-4b69-92a1-854614113206","Type":"ContainerDied","Data":"acb2519768f4a8cbd95d9eaf2ca7cf1d6c780654d921d2e93a41a8ddb3c42983"} Oct 29 21:56:12.985830 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:12.985791 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" podStartSLOduration=1.985779352 podStartE2EDuration="1.985779352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:12.985157064 +0000 UTC m=+834.727087737" watchObservedRunningTime="2026-10-29 21:56:12.985779352 +0000 UTC m=+834.727710016" Oct 29 21:56:13.949400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:13.949362 4543 generic.go:356] "Generic (PLEG): container finished" podID="8f3bc952-bbf9-4c61-b83f-cf4d65acf471" containerID="8e69bc6a48e0e216decc518c00769b7d0f97053c82ef4e2a93f3231bc894ce79" exitCode=0 Oct 29 21:56:13.949810 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:13.949411 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" event={"ID":"8f3bc952-bbf9-4c61-b83f-cf4d65acf471","Type":"ContainerDied","Data":"8e69bc6a48e0e216decc518c00769b7d0f97053c82ef4e2a93f3231bc894ce79"} Oct 29 21:56:14.100821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.100798 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:14.259759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.259735 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\" (UniqueName: \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\") pod \"f364c227-13d2-4b69-92a1-854614113206\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " Oct 29 21:56:14.259888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.259832 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\") pod \"f364c227-13d2-4b69-92a1-854614113206\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " Oct 29 21:56:14.259958 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.259941 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\") pod \"f364c227-13d2-4b69-92a1-854614113206\" (UID: \"f364c227-13d2-4b69-92a1-854614113206\") " Oct 29 21:56:14.260289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.260264 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca" pod "f364c227-13d2-4b69-92a1-854614113206" (UID: "f364c227-13d2-4b69-92a1-854614113206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:56:14.260500 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.260335 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364c227-13d2-4b69-92a1-854614113206-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:14.261926 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.261908 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca" pod "f364c227-13d2-4b69-92a1-854614113206" (UID: "f364c227-13d2-4b69-92a1-854614113206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:56:14.262067 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.262051 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w" pod "f364c227-13d2-4b69-92a1-854614113206" (UID: "f364c227-13d2-4b69-92a1-854614113206"). InnerVolumeSpecName "kube-api-access-54c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:56:14.361035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.361007 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54c2w\" (UniqueName: \"kubernetes.io/projected/f364c227-13d2-4b69-92a1-854614113206-kube-api-access-54c2w\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:14.361035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.361031 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364c227-13d2-4b69-92a1-854614113206-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:14.410345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.410317 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 21:56:14.410855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.410841 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f364c227-13d2-4b69-92a1-854614113206" containerName="revoke-keys" Oct 29 21:56:14.410905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.410857 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f364c227-13d2-4b69-92a1-854614113206" containerName="revoke-keys" Oct 29 21:56:14.410947 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.410938 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f364c227-13d2-4b69-92a1-854614113206" containerName="revoke-keys" Oct 29 21:56:14.414249 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.414232 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.432449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.432430 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"openshift-service-ca.crt\"" Oct 29 21:56:14.435051 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.435034 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 21:56:14.445296 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.445271 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"kube-root-ca.crt\"" Oct 29 21:56:14.445536 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.445518 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"default-dockercfg-58jq5\"" Oct 29 21:56:14.445628 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.445598 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"distinct-w1-29c2f2-kserve-self-signed-certs\"" Oct 29 21:56:14.562890 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.562841 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.562988 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.562894 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.563047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.563023 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.563126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.563096 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.563195 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.563169 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.563248 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.563198 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j9nq\" (UniqueName: \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.663736 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663703 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.663906 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663754 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.663906 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.663906 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663799 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j9nq\" (UniqueName: \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.664040 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663903 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.664040 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.663972 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.664251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.664225 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.664325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.664253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.664325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.664310 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.666453 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.666429 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.666973 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.666952 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.690271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.690248 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j9nq\" (UniqueName: \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\") pod \"distinct-w1-29c2f2-kserve-7f859bbf88-twnxw\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.725805 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.725784 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:14.958421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.958343 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" Oct 29 21:56:14.958421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.958384 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt" event={"ID":"f364c227-13d2-4b69-92a1-854614113206","Type":"ContainerDied","Data":"6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6"} Oct 29 21:56:14.958938 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:14.958457 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6275681cc05abb088003764505cc0d46664b24ba26e09755d817f68d1fb81de6" Oct 29 21:56:15.041224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.041199 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 21:56:15.042505 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:56:15.042463 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc0bac31_3a74_47eb_9106_8096a4440008.slice/crio-f07cb4b0008c05f850ff5a5f80a6c9af0c31794021aae494741f056d5fd616cb WatchSource:0}: Error finding container f07cb4b0008c05f850ff5a5f80a6c9af0c31794021aae494741f056d5fd616cb: Status 404 returned error can't find the container with id f07cb4b0008c05f850ff5a5f80a6c9af0c31794021aae494741f056d5fd616cb Oct 29 21:56:15.108438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.108414 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:15.279126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.270423 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\") pod \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " Oct 29 21:56:15.279126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.270541 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\" (UniqueName: \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\") pod \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " Oct 29 21:56:15.279126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.270583 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\") pod \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\" (UID: \"8f3bc952-bbf9-4c61-b83f-cf4d65acf471\") " Oct 29 21:56:15.279126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.274980 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca" pod "8f3bc952-bbf9-4c61-b83f-cf4d65acf471" (UID: "8f3bc952-bbf9-4c61-b83f-cf4d65acf471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:56:15.283658 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.280456 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh" pod "8f3bc952-bbf9-4c61-b83f-cf4d65acf471" (UID: "8f3bc952-bbf9-4c61-b83f-cf4d65acf471"). InnerVolumeSpecName "kube-api-access-hppgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:56:15.283658 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.283314 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca" pod "8f3bc952-bbf9-4c61-b83f-cf4d65acf471" (UID: "8f3bc952-bbf9-4c61-b83f-cf4d65acf471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:56:15.372480 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.372444 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:15.372646 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.372486 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:15.372646 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.372502 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hppgh\" (UniqueName: \"kubernetes.io/projected/8f3bc952-bbf9-4c61-b83f-cf4d65acf471-kube-api-access-hppgh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:56:15.972281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.972234 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" event={"ID":"8f3bc952-bbf9-4c61-b83f-cf4d65acf471","Type":"ContainerDied","Data":"8ff61579227b8d28206f2acfa26a05322de5cb95b686bfaa4defe880885689b8"} Oct 29 21:56:15.972855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.972826 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff61579227b8d28206f2acfa26a05322de5cb95b686bfaa4defe880885689b8" Oct 29 21:56:15.973648 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.973562 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns" Oct 29 21:56:15.975498 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.975464 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" event={"ID":"cc0bac31-3a74-47eb-9106-8096a4440008","Type":"ContainerStarted","Data":"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18"} Oct 29 21:56:15.975616 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:15.975509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" event={"ID":"cc0bac31-3a74-47eb-9106-8096a4440008","Type":"ContainerStarted","Data":"f07cb4b0008c05f850ff5a5f80a6c9af0c31794021aae494741f056d5fd616cb"} Oct 29 21:56:16.025050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:16.025003 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podStartSLOduration=2.024988123 podStartE2EDuration="2.024988123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:16.022590725 +0000 UTC m=+837.764521399" watchObservedRunningTime="2026-10-29 21:56:16.024988123 +0000 UTC m=+837.766918796" Oct 29 21:56:16.726335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:16.726303 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:19.298217 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.298176 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 21:56:19.298899 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.298876 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8f3bc952-bbf9-4c61-b83f-cf4d65acf471" containerName="revoke-keys" Oct 29 21:56:19.299003 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.298900 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3bc952-bbf9-4c61-b83f-cf4d65acf471" containerName="revoke-keys" Oct 29 21:56:19.299062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.299028 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3bc952-bbf9-4c61-b83f-cf4d65acf471" containerName="revoke-keys" Oct 29 21:56:19.305064 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.305036 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.325262 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.325235 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 21:56:19.329767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.329736 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86j5f\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.329896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.329847 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.329960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.329911 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330012 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330133 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330116 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330311 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330287 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330362 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330327 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330398 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.330486 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.330469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.335803 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.335781 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cea94a-openshift-default-dockercfg-56p6l\"" Oct 29 21:56:19.431612 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431587 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431638 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431660 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431676 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431719 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431701 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431724 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431751 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86j5f\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431795 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.431916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.431839 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.432176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.432152 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.432328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.432305 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.432398 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.432320 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.432463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.432397 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.432675 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.432541 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.434181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.434157 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.434337 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.434323 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.461457 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.461432 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86j5f\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.461602 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.461541 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.619976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.619928 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:19.945896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.945874 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 21:56:19.947666 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:56:19.947635 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e06dcf5_9c2a_4692_aa04_cfc2bcf05cd6.slice/crio-5a7fdc3632089f563cc4cbb704f7fe0d63912f11dbc8883be93f6d08f80818da WatchSource:0}: Error finding container 5a7fdc3632089f563cc4cbb704f7fe0d63912f11dbc8883be93f6d08f80818da: Status 404 returned error can't find the container with id 5a7fdc3632089f563cc4cbb704f7fe0d63912f11dbc8883be93f6d08f80818da Oct 29 21:56:19.950047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.950015 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:19.950170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.950099 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:19.950170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.950134 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:19.998642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:19.998617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" event={"ID":"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6","Type":"ContainerStarted","Data":"5a7fdc3632089f563cc4cbb704f7fe0d63912f11dbc8883be93f6d08f80818da"} Oct 29 21:56:21.006814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:21.006774 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" event={"ID":"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6","Type":"ContainerStarted","Data":"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f"} Oct 29 21:56:21.095771 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:21.095711 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podStartSLOduration=2.095692097 podStartE2EDuration="2.095692097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:21.079889192 +0000 UTC m=+842.821819867" watchObservedRunningTime="2026-10-29 21:56:21.095692097 +0000 UTC m=+842.837622798" Oct 29 21:56:21.620773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:21.620744 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:21.622509 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:21.622476 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:56:21.622648 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:21.622528 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 21:56:22.620823 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:22.620785 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:56:22.621237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:22.620841 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 21:56:23.621177 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:23.621140 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:56:23.621573 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:23.621204 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 21:56:24.620786 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:24.620752 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:56:24.620969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:24.620804 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 21:56:24.726470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:24.726445 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:24.766421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:24.766397 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:25.064499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:25.064471 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:56:25.620759 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:25.620719 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 21:56:25.620955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:25.620784 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 21:56:26.624707 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:26.624679 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:26.625199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:26.625096 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:26.625740 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:26.625718 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 21:56:27.959858 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:56:27.959820 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:56:46.556665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.556592 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 21:56:46.561279 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.561258 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.586151 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.586127 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 21:56:46.720058 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720027 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720184 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720068 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720187 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720240 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720320 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spggw\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720453 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720503 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720486 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.720552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.720540 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.821918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.821843 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.821918 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.821885 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.821925 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.821978 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822017 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822067 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822131 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822156 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spggw\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822345 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822363 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822664 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.822966 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.822938 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.824751 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.824718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.825031 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.825012 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.848144 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.848111 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.848253 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.848165 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spggw\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:46.900787 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:46.900761 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:47.225902 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:47.225881 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 21:56:47.227471 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:56:47.227445 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91175bfe_bfc9_4318_9e64_25bdfec8ab33.slice/crio-2bd5c2aae535edfc9c08ebe27a57cf33dbb1196c7782a2638bdc7062e10a0d9f WatchSource:0}: Error finding container 2bd5c2aae535edfc9c08ebe27a57cf33dbb1196c7782a2638bdc7062e10a0d9f: Status 404 returned error can't find the container with id 2bd5c2aae535edfc9c08ebe27a57cf33dbb1196c7782a2638bdc7062e10a0d9f Oct 29 21:56:47.229775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:47.229741 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:47.229894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:47.229815 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:47.229894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:47.229860 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:56:48.204491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.204446 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" event={"ID":"91175bfe-bfc9-4318-9e64-25bdfec8ab33","Type":"ContainerStarted","Data":"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b"} Oct 29 21:56:48.204885 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.204500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" event={"ID":"91175bfe-bfc9-4318-9e64-25bdfec8ab33","Type":"ContainerStarted","Data":"2bd5c2aae535edfc9c08ebe27a57cf33dbb1196c7782a2638bdc7062e10a0d9f"} Oct 29 21:56:48.276617 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.276561 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podStartSLOduration=2.276541329 podStartE2EDuration="2.276541329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:48.274398144 +0000 UTC m=+870.016328817" watchObservedRunningTime="2026-10-29 21:56:48.276541329 +0000 UTC m=+870.018472002" Oct 29 21:56:48.901863 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.901824 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:48.903137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.903111 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:48.903262 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:48.903160 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:49.901341 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:49.901308 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:49.901728 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:49.901359 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:50.901796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:50.901763 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:50.902197 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:50.901815 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:51.901068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:51.901033 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:51.901265 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:51.901095 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:52.901235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:52.901204 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:52.901623 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:52.901256 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:53.901717 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:53.901678 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:53.902211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:53.901738 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 21:56:54.906293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:54.906262 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:54.906712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:54.906684 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:54.907479 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:54.907456 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 21:56:56.023347 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.023315 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 21:56:56.030819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.030792 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.048400 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.048371 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 21:56:56.057382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.057356 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"distinct-w4-cea94a-kserve-self-signed-certs\"" Oct 29 21:56:56.149134 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149108 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.149286 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149270 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.149330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.149377 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149333 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.149481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149427 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.149481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.149458 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hm7\" (UniqueName: \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250367 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250381 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250417 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250487 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250531 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hm7\" (UniqueName: \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.250819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.250573 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.251468 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.251439 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.252106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.252060 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.252260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.252112 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.257594 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.257508 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.267733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.267673 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.278104 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.278026 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hm7\" (UniqueName: \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\") pod \"distinct-w4-cea94a-kserve-7679676bd-g5zlv\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.348550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.348523 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:56.656006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:56.655982 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 21:56:56.657696 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:56:56.657669 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1944fc6c_9357_4c09_af34_46839f8ac52a.slice/crio-996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6 WatchSource:0}: Error finding container 996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6: Status 404 returned error can't find the container with id 996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6 Oct 29 21:56:57.268381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:57.268343 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" event={"ID":"1944fc6c-9357-4c09-af34-46839f8ac52a","Type":"ContainerStarted","Data":"fbf1732006c310adfef15fdf46ec58b7e57734a51eba9a4ea56af8acf778fb6d"} Oct 29 21:56:57.268754 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:57.268389 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" event={"ID":"1944fc6c-9357-4c09-af34-46839f8ac52a","Type":"ContainerStarted","Data":"996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6"} Oct 29 21:56:57.313793 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:57.313747 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podStartSLOduration=2.313734039 podStartE2EDuration="2.313734039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:57.313152727 +0000 UTC m=+879.055083401" watchObservedRunningTime="2026-10-29 21:56:57.313734039 +0000 UTC m=+879.055664734" Oct 29 21:56:57.348963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:56:57.348925 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:56:57.961354 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:56:57.961318 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:57:01.718702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.718669 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 21:57:01.723035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.723017 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.763992 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.763965 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 21:57:01.808119 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808049 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808417 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808588 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808656 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808710 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808754 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808849 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808799 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.808927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808906 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.809017 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.808996 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8j8\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910530 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910623 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910651 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910665 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910683 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.910747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910708 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8j8\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.910878 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.911061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911274 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.911111 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911274 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.911179 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.911380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.911335 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.912004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.911977 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.913423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.913399 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.913586 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.913566 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.936389 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.936367 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:01.936542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:01.936525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8j8\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:02.036855 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:02.036799 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:02.368188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:02.368163 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 21:57:02.369639 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:02.369610 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b62808_e524_42d2_b71b_01f7b2e51c2a.slice/crio-483cda0bbd107f903f05659e34f28a8922b0609aa8f6d4b9ff9224fba0dbbaae WatchSource:0}: Error finding container 483cda0bbd107f903f05659e34f28a8922b0609aa8f6d4b9ff9224fba0dbbaae: Status 404 returned error can't find the container with id 483cda0bbd107f903f05659e34f28a8922b0609aa8f6d4b9ff9224fba0dbbaae Oct 29 21:57:02.371835 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:02.371803 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:02.371945 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:02.371877 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:02.371945 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:02.371923 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:03.305543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:03.305509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" event={"ID":"d7b62808-e524-42d2-b71b-01f7b2e51c2a","Type":"ContainerStarted","Data":"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf"} Oct 29 21:57:03.305963 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:03.305565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" event={"ID":"d7b62808-e524-42d2-b71b-01f7b2e51c2a","Type":"ContainerStarted","Data":"483cda0bbd107f903f05659e34f28a8922b0609aa8f6d4b9ff9224fba0dbbaae"} Oct 29 21:57:03.387218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:03.387168 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podStartSLOduration=2.387155018 podStartE2EDuration="2.387155018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:03.372538613 +0000 UTC m=+885.114469287" watchObservedRunningTime="2026-10-29 21:57:03.387155018 +0000 UTC m=+885.129085691" Oct 29 21:57:04.046004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:04.040312 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:04.050099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:04.048163 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 21:57:04.050099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:04.048225 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 21:57:05.042554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:05.042504 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 21:57:05.043199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:05.043162 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 21:57:06.047102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:06.042935 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 21:57:06.047102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:06.042998 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 21:57:06.353388 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:06.353287 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:57:06.366038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:06.366009 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:57:07.044101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:07.038151 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 21:57:07.044101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:07.038209 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 21:57:07.364447 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:07.364371 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:57:08.041102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:08.037382 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 21:57:08.041102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:08.037442 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 21:57:09.261335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:09.261305 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:09.262451 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:09.262431 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:09.263013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:09.262992 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 21:57:27.112666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.112634 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 21:57:27.118301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.118275 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.140314 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.140283 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 21:57:27.148862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.148837 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"openshift-service-ca.crt\"" Oct 29 21:57:27.148997 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.148860 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"kube-root-ca.crt\"" Oct 29 21:57:27.149057 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.149009 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"default-dockercfg-x7kjc\"" Oct 29 21:57:27.149154 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.149132 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"distinct-2-w6-863f1b-kserve-self-signed-certs\"" Oct 29 21:57:27.253626 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253592 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.253749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253668 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.253811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253792 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5qp\" (UniqueName: \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.253856 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253826 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.253856 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253848 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.253967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.253918 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355171 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355146 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5qp\" (UniqueName: \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355185 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355208 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355233 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355360 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355423 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355634 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355614 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355641 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.355827 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.355809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.357583 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.357560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.357848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.357829 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.382775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.382696 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5qp\" (UniqueName: \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\") pod \"distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.432996 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.432932 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:27.748394 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:27.748372 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 21:57:27.750044 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:27.750020 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c8048b_cb51_4f60_81b2_ed9dab8f8a06.slice/crio-9953f19fcec512e322aa25d3c9eb4b0fab9910b27ad828dcb6ae8dd4c21c55e4 WatchSource:0}: Error finding container 9953f19fcec512e322aa25d3c9eb4b0fab9910b27ad828dcb6ae8dd4c21c55e4: Status 404 returned error can't find the container with id 9953f19fcec512e322aa25d3c9eb4b0fab9910b27ad828dcb6ae8dd4c21c55e4 Oct 29 21:57:27.958950 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:57:27.958919 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:57:28.468985 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:28.468954 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" event={"ID":"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06","Type":"ContainerStarted","Data":"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7"} Oct 29 21:57:28.469356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:28.468994 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" event={"ID":"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06","Type":"ContainerStarted","Data":"9953f19fcec512e322aa25d3c9eb4b0fab9910b27ad828dcb6ae8dd4c21c55e4"} Oct 29 21:57:28.517687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:28.517637 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podStartSLOduration=1.517624555 podStartE2EDuration="1.517624555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:28.513647789 +0000 UTC m=+910.255578462" watchObservedRunningTime="2026-10-29 21:57:28.517624555 +0000 UTC m=+910.259555273" Oct 29 21:57:29.433807 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:29.433780 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:31.518376 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.518348 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz"] Oct 29 21:57:31.522631 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.522611 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.545696 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.545660 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz"] Oct 29 21:57:31.552765 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.552741 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:57:31.552919 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.552761 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:57:31.553062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.552863 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:57:31.553580 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.553560 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:57:31.606655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.606628 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.606808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.606663 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.606877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.606849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srn5c\" (UniqueName: \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.708186 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.708160 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn5c\" (UniqueName: \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.708327 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.708265 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.708327 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.708288 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.708928 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.708909 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.710786 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.710770 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.734745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.734724 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn5c\" (UniqueName: \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\") pod \"maas-api-revoke-sub-ba698a2c-7zcgz\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:31.837597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:31.837547 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:32.143112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:32.143069 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz"] Oct 29 21:57:32.144630 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:32.144587 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0179a221_f690_4f0b_b3cc_81d1cfa03b4f.slice/crio-5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0 WatchSource:0}: Error finding container 5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0: Status 404 returned error can't find the container with id 5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0 Oct 29 21:57:32.500461 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:32.500426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" event={"ID":"0179a221-f690-4f0b-b3cc-81d1cfa03b4f","Type":"ContainerStarted","Data":"29f48d365a2495616c264cac5159748855f413ed9a9fecefea5d57b1c9fcf9e1"} Oct 29 21:57:32.500582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:32.500474 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" event={"ID":"0179a221-f690-4f0b-b3cc-81d1cfa03b4f","Type":"ContainerStarted","Data":"5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0"} Oct 29 21:57:32.545812 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:32.545773 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" podStartSLOduration=1.545758968 podStartE2EDuration="1.545758968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:32.54319826 +0000 UTC m=+914.285128933" watchObservedRunningTime="2026-10-29 21:57:32.545758968 +0000 UTC m=+914.287689683" Oct 29 21:57:33.509426 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:33.509390 4543 generic.go:356] "Generic (PLEG): container finished" podID="0179a221-f690-4f0b-b3cc-81d1cfa03b4f" containerID="29f48d365a2495616c264cac5159748855f413ed9a9fecefea5d57b1c9fcf9e1" exitCode=0 Oct 29 21:57:33.509585 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:33.509481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" event={"ID":"0179a221-f690-4f0b-b3cc-81d1cfa03b4f","Type":"ContainerDied","Data":"29f48d365a2495616c264cac5159748855f413ed9a9fecefea5d57b1c9fcf9e1"} Oct 29 21:57:34.660792 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.660769 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:34.741155 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.741131 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\") pod \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " Oct 29 21:57:34.741293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.741186 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\" (UniqueName: \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\") pod \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " Oct 29 21:57:34.741293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.741222 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\") pod \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\" (UID: \"0179a221-f690-4f0b-b3cc-81d1cfa03b4f\") " Oct 29 21:57:34.741565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.741524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca" pod "0179a221-f690-4f0b-b3cc-81d1cfa03b4f" (UID: "0179a221-f690-4f0b-b3cc-81d1cfa03b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:57:34.741687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.741631 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:34.743450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.743418 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca" pod "0179a221-f690-4f0b-b3cc-81d1cfa03b4f" (UID: "0179a221-f690-4f0b-b3cc-81d1cfa03b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:34.743568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.743528 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c" pod "0179a221-f690-4f0b-b3cc-81d1cfa03b4f" (UID: "0179a221-f690-4f0b-b3cc-81d1cfa03b4f"). InnerVolumeSpecName "kube-api-access-srn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:34.842277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.842226 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srn5c\" (UniqueName: \"kubernetes.io/projected/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-kube-api-access-srn5c\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:34.842277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:34.842248 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0179a221-f690-4f0b-b3cc-81d1cfa03b4f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:35.522988 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.522961 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" Oct 29 21:57:35.523149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.522962 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz" event={"ID":"0179a221-f690-4f0b-b3cc-81d1cfa03b4f","Type":"ContainerDied","Data":"5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0"} Oct 29 21:57:35.523149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.523040 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c46ed5e7401d23c0ea5124c80a371d33691dabf3cd1088e5fdb333df19659a0" Oct 29 21:57:35.619879 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.619852 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c"] Oct 29 21:57:35.620463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.620448 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0179a221-f690-4f0b-b3cc-81d1cfa03b4f" containerName="revoke-keys" Oct 29 21:57:35.620526 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.620465 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0179a221-f690-4f0b-b3cc-81d1cfa03b4f" containerName="revoke-keys" Oct 29 21:57:35.620588 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.620579 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0179a221-f690-4f0b-b3cc-81d1cfa03b4f" containerName="revoke-keys" Oct 29 21:57:35.624147 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.624133 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.641824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.641800 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:57:35.641955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.641803 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:57:35.642937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.642917 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c"] Oct 29 21:57:35.652894 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.652871 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:57:35.653052 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.653009 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:57:35.750578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.750545 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.750969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.750670 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.750969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.750731 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7dcf\" (UniqueName: \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.851845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.851780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.851845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.851819 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7dcf\" (UniqueName: \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.852011 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.851975 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.852587 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.852565 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.854250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.854221 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.877693 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.877671 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7dcf\" (UniqueName: \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\") pod \"maas-api-revoke-sub-e4028a4e-4jw8c\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:35.938416 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:35.938395 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:36.242702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:36.242672 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c"] Oct 29 21:57:36.245881 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:36.245850 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c727f81_937b_46c6_84d3_71d0efde3419.slice/crio-2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916 WatchSource:0}: Error finding container 2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916: Status 404 returned error can't find the container with id 2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916 Oct 29 21:57:36.532021 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:36.531991 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" event={"ID":"9c727f81-937b-46c6-84d3-71d0efde3419","Type":"ContainerStarted","Data":"2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916"} Oct 29 21:57:37.433269 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:37.433216 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:37.445756 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:37.445737 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:37.539713 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:37.539686 4543 generic.go:356] "Generic (PLEG): container finished" podID="9c727f81-937b-46c6-84d3-71d0efde3419" containerID="406a8dc1b429eeaff172a1ab455943602b219d694d69ba29fc54769c537c3671" exitCode=0 Oct 29 21:57:37.539825 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:37.539766 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" event={"ID":"9c727f81-937b-46c6-84d3-71d0efde3419","Type":"ContainerDied","Data":"406a8dc1b429eeaff172a1ab455943602b219d694d69ba29fc54769c537c3671"} Oct 29 21:57:37.551016 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:37.550996 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:57:38.711282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.711254 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:38.884672 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.884587 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\") pod \"9c727f81-937b-46c6-84d3-71d0efde3419\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " Oct 29 21:57:38.884827 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.884731 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\" (UniqueName: \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\") pod \"9c727f81-937b-46c6-84d3-71d0efde3419\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " Oct 29 21:57:38.884898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.884864 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\") pod \"9c727f81-937b-46c6-84d3-71d0efde3419\" (UID: \"9c727f81-937b-46c6-84d3-71d0efde3419\") " Oct 29 21:57:38.885247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.885222 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca" pod "9c727f81-937b-46c6-84d3-71d0efde3419" (UID: "9c727f81-937b-46c6-84d3-71d0efde3419"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:57:38.886777 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.886756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca" pod "9c727f81-937b-46c6-84d3-71d0efde3419" (UID: "9c727f81-937b-46c6-84d3-71d0efde3419"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:38.886845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.886799 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf" pod "9c727f81-937b-46c6-84d3-71d0efde3419" (UID: "9c727f81-937b-46c6-84d3-71d0efde3419"). InnerVolumeSpecName "kube-api-access-t7dcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:38.986481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.986456 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:38.986481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.986480 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7dcf\" (UniqueName: \"kubernetes.io/projected/9c727f81-937b-46c6-84d3-71d0efde3419-kube-api-access-t7dcf\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:38.986598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:38.986491 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c727f81-937b-46c6-84d3-71d0efde3419-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:39.554859 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:39.554833 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" Oct 29 21:57:39.555051 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:39.554870 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c" event={"ID":"9c727f81-937b-46c6-84d3-71d0efde3419","Type":"ContainerDied","Data":"2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916"} Oct 29 21:57:39.555051 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:39.554929 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2806e7b389550db474db67be3afc724f239e382d9d98f0a7d4c1ab5fc0aef916" Oct 29 21:57:39.605730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:39.605703 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:57:41.929999 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.929964 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg"] Oct 29 21:57:41.930862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.930843 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9c727f81-937b-46c6-84d3-71d0efde3419" containerName="revoke-keys" Oct 29 21:57:41.930920 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.930867 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c727f81-937b-46c6-84d3-71d0efde3419" containerName="revoke-keys" Oct 29 21:57:41.931062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.931048 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c727f81-937b-46c6-84d3-71d0efde3419" containerName="revoke-keys" Oct 29 21:57:41.935198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.935179 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:41.952920 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.952897 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg"] Oct 29 21:57:41.963148 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.963128 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:57:41.963271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.963146 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:57:41.998249 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.998228 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:57:41.998334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:41.998272 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:57:42.123012 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.122982 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.123179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.123015 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5pvp\" (UniqueName: \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.123276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.123262 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.224848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.224824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.225004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.224882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.225004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.224992 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pvp\" (UniqueName: \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.225583 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.225560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.227271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.227248 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.250623 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.250597 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pvp\" (UniqueName: \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\") pod \"maas-api-revoke-sub-096719ca-cvdpg\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.547814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.547746 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:42.854937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:42.854904 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg"] Oct 29 21:57:42.855129 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:42.855058 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2964a666_ab61_41e7_b152_8ec68408bd5d.slice/crio-71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6 WatchSource:0}: Error finding container 71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6: Status 404 returned error can't find the container with id 71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6 Oct 29 21:57:43.585161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:43.585117 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" event={"ID":"2964a666-ab61-41e7-b152-8ec68408bd5d","Type":"ContainerStarted","Data":"b02b80154c00993d06202dbb20e077ac93d536b40b3e17d721dc3fce0fce4c16"} Oct 29 21:57:43.585594 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:43.585193 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" event={"ID":"2964a666-ab61-41e7-b152-8ec68408bd5d","Type":"ContainerStarted","Data":"71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6"} Oct 29 21:57:43.639766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:43.639716 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" podStartSLOduration=2.639701056 podStartE2EDuration="2.639701056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:43.639374257 +0000 UTC m=+925.381304929" watchObservedRunningTime="2026-10-29 21:57:43.639701056 +0000 UTC m=+925.381631728" Oct 29 21:57:44.593026 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:44.592992 4543 generic.go:356] "Generic (PLEG): container finished" podID="2964a666-ab61-41e7-b152-8ec68408bd5d" containerID="b02b80154c00993d06202dbb20e077ac93d536b40b3e17d721dc3fce0fce4c16" exitCode=0 Oct 29 21:57:44.593412 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:44.593090 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" event={"ID":"2964a666-ab61-41e7-b152-8ec68408bd5d","Type":"ContainerDied","Data":"b02b80154c00993d06202dbb20e077ac93d536b40b3e17d721dc3fce0fce4c16"} Oct 29 21:57:45.746910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.746882 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:45.766948 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.766921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\") pod \"2964a666-ab61-41e7-b152-8ec68408bd5d\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " Oct 29 21:57:45.767060 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.766962 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\") pod \"2964a666-ab61-41e7-b152-8ec68408bd5d\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " Oct 29 21:57:45.767162 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.767101 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\" (UniqueName: \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\") pod \"2964a666-ab61-41e7-b152-8ec68408bd5d\" (UID: \"2964a666-ab61-41e7-b152-8ec68408bd5d\") " Oct 29 21:57:45.767313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.767287 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca" pod "2964a666-ab61-41e7-b152-8ec68408bd5d" (UID: "2964a666-ab61-41e7-b152-8ec68408bd5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:57:45.767609 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.767585 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:45.772577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.772531 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp" pod "2964a666-ab61-41e7-b152-8ec68408bd5d" (UID: "2964a666-ab61-41e7-b152-8ec68408bd5d"). InnerVolumeSpecName "kube-api-access-d5pvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:45.780864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.780826 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca" pod "2964a666-ab61-41e7-b152-8ec68408bd5d" (UID: "2964a666-ab61-41e7-b152-8ec68408bd5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:45.868687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.868607 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964a666-ab61-41e7-b152-8ec68408bd5d-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:45.868687 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:45.868635 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5pvp\" (UniqueName: \"kubernetes.io/projected/2964a666-ab61-41e7-b152-8ec68408bd5d-kube-api-access-d5pvp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:46.606745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.606716 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" Oct 29 21:57:46.606926 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.606772 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg" event={"ID":"2964a666-ab61-41e7-b152-8ec68408bd5d","Type":"ContainerDied","Data":"71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6"} Oct 29 21:57:46.606926 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.606820 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71b9b0d4cd07c1f427f27c666d10dfc7f4a01f7de7d619cbfad367773feb7ec6" Oct 29 21:57:46.888038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.888010 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 21:57:46.888565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.888552 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2964a666-ab61-41e7-b152-8ec68408bd5d" containerName="revoke-keys" Oct 29 21:57:46.888608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.888566 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2964a666-ab61-41e7-b152-8ec68408bd5d" containerName="revoke-keys" Oct 29 21:57:46.888666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.888657 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2964a666-ab61-41e7-b152-8ec68408bd5d" containerName="revoke-keys" Oct 29 21:57:46.892255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.892231 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.927892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.927868 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 21:57:46.980578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980550 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.980731 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980605 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.980810 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980733 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.980810 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980778 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.980919 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980894 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.980998 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.980977 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.981117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.981024 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.981181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.981154 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:46.981231 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:46.981183 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.081901 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.081877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082034 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.081917 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082034 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.081964 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082034 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.081990 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082035 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082147 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082189 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082213 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082326 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082413 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082516 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082622 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.082896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.082878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.084459 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.084438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.084684 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.084667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.109947 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.109920 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.110477 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.110458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.207355 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.207296 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:47.530566 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.530537 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 21:57:47.530755 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:47.530732 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cf2d34_422f_4e62_83b5_c53ac1eabbfb.slice/crio-42c4ae6c24d9dd95e1d9a23a99966910a191e8dd1f3fbc6c760a11ef55118eca WatchSource:0}: Error finding container 42c4ae6c24d9dd95e1d9a23a99966910a191e8dd1f3fbc6c760a11ef55118eca: Status 404 returned error can't find the container with id 42c4ae6c24d9dd95e1d9a23a99966910a191e8dd1f3fbc6c760a11ef55118eca Oct 29 21:57:47.533150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.533115 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:47.533246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.533179 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:47.533246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.533211 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:57:47.617161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.617119 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" event={"ID":"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb","Type":"ContainerStarted","Data":"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7"} Oct 29 21:57:47.617287 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.617198 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" event={"ID":"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb","Type":"ContainerStarted","Data":"42c4ae6c24d9dd95e1d9a23a99966910a191e8dd1f3fbc6c760a11ef55118eca"} Oct 29 21:57:47.665857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:47.665793 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podStartSLOduration=1.665774661 podStartE2EDuration="1.665774661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:47.665175007 +0000 UTC m=+929.407105681" watchObservedRunningTime="2026-10-29 21:57:47.665774661 +0000 UTC m=+929.407705337" Oct 29 21:57:48.217101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:48.213141 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:48.221391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:48.220288 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:48.221391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:48.220369 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:49.207813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:49.207778 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:49.207989 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:49.207841 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:50.207646 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.207605 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:50.208046 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.207670 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:50.965293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.965260 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b"] Oct 29 21:57:50.969426 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.969403 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:50.988473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.988447 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b"] Oct 29 21:57:50.998482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.998458 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:57:50.998482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.998464 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:57:50.998645 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:50.998524 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:57:51.000046 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.000026 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:57:51.027619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.027595 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.027726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.027713 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbzm\" (UniqueName: \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.027783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.027758 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.153798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.153767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbzm\" (UniqueName: \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.153955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.153816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.153955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.153930 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.154660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.154635 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.156304 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.156281 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.179553 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.179522 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbzm\" (UniqueName: \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\") pod \"maas-api-revoke-sub-5a6a4236-pfq2b\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.208340 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.208310 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:51.208681 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.208392 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:51.282998 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.282936 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:51.613790 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.613763 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b"] Oct 29 21:57:51.614600 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:51.614570 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3763038c_fb3b_43a6_b280_5d12cfee2052.slice/crio-566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a WatchSource:0}: Error finding container 566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a: Status 404 returned error can't find the container with id 566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a Oct 29 21:57:51.666824 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:51.666794 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" event={"ID":"3763038c-fb3b-43a6-b280-5d12cfee2052","Type":"ContainerStarted","Data":"566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a"} Oct 29 21:57:52.207830 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:52.207757 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:52.207830 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:52.207804 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:52.675502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:52.675469 4543 generic.go:356] "Generic (PLEG): container finished" podID="3763038c-fb3b-43a6-b280-5d12cfee2052" containerID="429974dbc27f0a3ca8da94de7747205e7d3ff9acbc4863e2cdff61e5a3c1bfa9" exitCode=0 Oct 29 21:57:52.675813 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:52.675544 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" event={"ID":"3763038c-fb3b-43a6-b280-5d12cfee2052","Type":"ContainerDied","Data":"429974dbc27f0a3ca8da94de7747205e7d3ff9acbc4863e2cdff61e5a3c1bfa9"} Oct 29 21:57:53.207857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.207822 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:53.208011 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.207870 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:53.899540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.899500 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:53.993192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.993136 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\" (UniqueName: \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\") pod \"3763038c-fb3b-43a6-b280-5d12cfee2052\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " Oct 29 21:57:53.993490 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.993466 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\") pod \"3763038c-fb3b-43a6-b280-5d12cfee2052\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " Oct 29 21:57:53.993847 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.993824 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\") pod \"3763038c-fb3b-43a6-b280-5d12cfee2052\" (UID: \"3763038c-fb3b-43a6-b280-5d12cfee2052\") " Oct 29 21:57:53.994875 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:53.994854 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca" pod "3763038c-fb3b-43a6-b280-5d12cfee2052" (UID: "3763038c-fb3b-43a6-b280-5d12cfee2052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:57:54.025666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.025636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm" pod "3763038c-fb3b-43a6-b280-5d12cfee2052" (UID: "3763038c-fb3b-43a6-b280-5d12cfee2052"). InnerVolumeSpecName "kube-api-access-tqbzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:54.045429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.045270 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca" pod "3763038c-fb3b-43a6-b280-5d12cfee2052" (UID: "3763038c-fb3b-43a6-b280-5d12cfee2052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:54.096543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.096261 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:54.096725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.096607 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbzm\" (UniqueName: \"kubernetes.io/projected/3763038c-fb3b-43a6-b280-5d12cfee2052-kube-api-access-tqbzm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:54.096725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.096628 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763038c-fb3b-43a6-b280-5d12cfee2052-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:54.211100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.208659 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:57:54.211100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.208722 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 29 21:57:54.634105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.627097 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerName="main" containerID="cri-o://9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1" gracePeriod=45 Oct 29 21:57:54.696892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.696859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" event={"ID":"3763038c-fb3b-43a6-b280-5d12cfee2052","Type":"ContainerDied","Data":"566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a"} Oct 29 21:57:54.697036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.696910 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="566f79a5427cf8451044f98e55b590ae1412d89642b8298f55b89f345462669a" Oct 29 21:57:54.697036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:54.696974 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b" Oct 29 21:57:55.173636 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.173286 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:57:55.371160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.370712 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.371160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.370802 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.371160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.370895 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.371160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.371025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.382350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.371237 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\" (UniqueName: \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.382350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.371293 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\" (UniqueName: \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\") pod \"850980f8-dd9e-4f43-95ea-bea0cab17f03\" (UID: \"850980f8-dd9e-4f43-95ea-bea0cab17f03\") " Oct 29 21:57:55.382350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.378146 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:55.382350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.379511 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:55.382350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.380259 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:55.392523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.392484 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:55.399562 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.399350 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:55.399562 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.399448 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q" pod "850980f8-dd9e-4f43-95ea-bea0cab17f03" (UID: "850980f8-dd9e-4f43-95ea-bea0cab17f03"). InnerVolumeSpecName "kube-api-access-t592q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:55.403412 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.403388 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:55.404795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.404774 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:55.405726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.405652 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.478931 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.478961 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850980f8-dd9e-4f43-95ea-bea0cab17f03-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.478975 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t592q\" (UniqueName: \"kubernetes.io/projected/850980f8-dd9e-4f43-95ea-bea0cab17f03-kube-api-access-t592q\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.478990 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.479005 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.490101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.479017 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850980f8-dd9e-4f43-95ea-bea0cab17f03-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:57:55.759112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.759056 4543 generic.go:356] "Generic (PLEG): container finished" podID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerID="9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1" exitCode=0 Oct 29 21:57:55.760260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.760239 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" Oct 29 21:57:55.766429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.766220 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" event={"ID":"850980f8-dd9e-4f43-95ea-bea0cab17f03","Type":"ContainerDied","Data":"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1"} Oct 29 21:57:55.766429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.766278 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" event={"ID":"850980f8-dd9e-4f43-95ea-bea0cab17f03","Type":"ContainerDied","Data":"e3f42fb43eeb3d8bc9d42e02b946112a1e28cb3846f1ca6b8f848ece03617baa"} Oct 29 21:57:55.766429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.766301 4543 scope.go:122] "RemoveContainer" containerID="9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1" Oct 29 21:57:55.920044 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.920016 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:57:55.923002 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.922976 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w"] Oct 29 21:57:55.923469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.923282 4543 scope.go:122] "RemoveContainer" containerID="9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1" Oct 29 21:57:55.923693 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:57:55.923676 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1\": container with ID starting with 9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1 not found: ID does not exist" containerID="9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1" Oct 29 21:57:55.923799 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.923780 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1"} err="failed to get container status \"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1\": rpc error: code = NotFound desc = could not find container \"9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1\": container with ID starting with 9efed04d65da106149b1153d6445444d5c79dc3da64b470c94590a0436fb09d1 not found: ID does not exist" Oct 29 21:57:55.954543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:55.954502 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-a-fb842e5c-kserve-568bbc96bd-fqf2w" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.598102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:56.587172 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" path="/var/lib/kubelet/pods/850980f8-dd9e-4f43-95ea-bea0cab17f03/volumes" Oct 29 21:57:57.960551 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:57:57.960516 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:57:59.048358 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.048323 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv"] Oct 29 21:57:59.049202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049181 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3763038c-fb3b-43a6-b280-5d12cfee2052" containerName="revoke-keys" Oct 29 21:57:59.049255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049205 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3763038c-fb3b-43a6-b280-5d12cfee2052" containerName="revoke-keys" Oct 29 21:57:59.049255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049236 4543 cpu_manager.go:414] "containerMap: removing container" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerName="main" Oct 29 21:57:59.049255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049244 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerName="main" Oct 29 21:57:59.049397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049385 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3763038c-fb3b-43a6-b280-5d12cfee2052" containerName="revoke-keys" Oct 29 21:57:59.049432 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.049401 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="850980f8-dd9e-4f43-95ea-bea0cab17f03" containerName="main" Oct 29 21:57:59.054427 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.054406 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.071112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.071088 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv"] Oct 29 21:57:59.081033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.081009 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:57:59.081033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.081014 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:57:59.081273 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.081095 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:57:59.084652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.084633 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:57:59.147384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.147349 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmpg\" (UniqueName: \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.147384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.147389 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.147609 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.147410 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.248192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.248165 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmpg\" (UniqueName: \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.248395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.248198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.248395 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.248218 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.248900 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.248880 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.250654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.250631 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.274059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.274004 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmpg\" (UniqueName: \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\") pod \"maas-api-revoke-sub-b489e3b3-nxjrv\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.371293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.371232 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:57:59.676789 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.676764 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv"] Oct 29 21:57:59.678613 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:57:59.678584 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2840dec5_b892_4231_a7f6_d5599d4d5a34.slice/crio-09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c WatchSource:0}: Error finding container 09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c: Status 404 returned error can't find the container with id 09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c Oct 29 21:57:59.795449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:57:59.795413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" event={"ID":"2840dec5-b892-4231-a7f6-d5599d4d5a34","Type":"ContainerStarted","Data":"09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c"} Oct 29 21:58:00.804666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:00.804589 4543 generic.go:356] "Generic (PLEG): container finished" podID="2840dec5-b892-4231-a7f6-d5599d4d5a34" containerID="0f85dcad1213621db552b158da3d7baa71a2857fa67189a0b4049068533e0bd0" exitCode=0 Oct 29 21:58:00.804666 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:00.804619 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" event={"ID":"2840dec5-b892-4231-a7f6-d5599d4d5a34","Type":"ContainerDied","Data":"0f85dcad1213621db552b158da3d7baa71a2857fa67189a0b4049068533e0bd0"} Oct 29 21:58:01.966065 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:01.966043 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:58:02.080232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.080173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\") pod \"2840dec5-b892-4231-a7f6-d5599d4d5a34\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " Oct 29 21:58:02.080232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.080210 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\") pod \"2840dec5-b892-4231-a7f6-d5599d4d5a34\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " Oct 29 21:58:02.080402 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.080338 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\" (UniqueName: \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\") pod \"2840dec5-b892-4231-a7f6-d5599d4d5a34\" (UID: \"2840dec5-b892-4231-a7f6-d5599d4d5a34\") " Oct 29 21:58:02.080563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.080539 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca" pod "2840dec5-b892-4231-a7f6-d5599d4d5a34" (UID: "2840dec5-b892-4231-a7f6-d5599d4d5a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:02.080707 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.080688 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:02.082614 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.082587 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca" pod "2840dec5-b892-4231-a7f6-d5599d4d5a34" (UID: "2840dec5-b892-4231-a7f6-d5599d4d5a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:02.082811 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.082788 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg" pod "2840dec5-b892-4231-a7f6-d5599d4d5a34" (UID: "2840dec5-b892-4231-a7f6-d5599d4d5a34"). InnerVolumeSpecName "kube-api-access-7lmpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:02.181795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.181772 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2840dec5-b892-4231-a7f6-d5599d4d5a34-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:02.181795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.181795 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmpg\" (UniqueName: \"kubernetes.io/projected/2840dec5-b892-4231-a7f6-d5599d4d5a34-kube-api-access-7lmpg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:02.832495 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.832459 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" Oct 29 21:58:02.832495 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.832475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv" event={"ID":"2840dec5-b892-4231-a7f6-d5599d4d5a34","Type":"ContainerDied","Data":"09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c"} Oct 29 21:58:02.832750 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:02.832524 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09717cfc4d6382b33cab7c1fb2a2af6f714dc6b3d404210498c087b064de8c8c" Oct 29 21:58:05.114533 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.114498 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s"] Oct 29 21:58:05.115150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.115134 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2840dec5-b892-4231-a7f6-d5599d4d5a34" containerName="revoke-keys" Oct 29 21:58:05.115213 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.115152 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2840dec5-b892-4231-a7f6-d5599d4d5a34" containerName="revoke-keys" Oct 29 21:58:05.115270 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.115261 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2840dec5-b892-4231-a7f6-d5599d4d5a34" containerName="revoke-keys" Oct 29 21:58:05.118542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.118528 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.144672 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.144636 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s"] Oct 29 21:58:05.152923 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.152899 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:58:05.153063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.152931 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:58:05.153063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.152933 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:58:05.153063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.152997 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:58:05.216300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.216277 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.216422 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.216315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvtb\" (UniqueName: \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.216475 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.216460 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.317086 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.317049 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.317202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.317190 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.317242 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.317232 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvtb\" (UniqueName: \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.317773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.317751 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.319668 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.319637 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.343070 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.343050 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvtb\" (UniqueName: \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\") pod \"maas-api-revoke-sub-67c650fe-rvv9s\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.442405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.442356 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:05.749608 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.749581 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s"] Oct 29 21:58:05.750235 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:05.750211 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd83061_3276_4968_996f_efe710794e22.slice/crio-949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd WatchSource:0}: Error finding container 949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd: Status 404 returned error can't find the container with id 949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd Oct 29 21:58:05.852764 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:05.852733 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" event={"ID":"1cd83061-3276-4968-996f-efe710794e22","Type":"ContainerStarted","Data":"949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd"} Oct 29 21:58:06.860403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:06.860338 4543 generic.go:356] "Generic (PLEG): container finished" podID="1cd83061-3276-4968-996f-efe710794e22" containerID="b7bab7bf7f19dd0c68d49049761747b93e8d64bc1f0c6a19e06f11755ad65f76" exitCode=0 Oct 29 21:58:06.860804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:06.860478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" event={"ID":"1cd83061-3276-4968-996f-efe710794e22","Type":"ContainerDied","Data":"b7bab7bf7f19dd0c68d49049761747b93e8d64bc1f0c6a19e06f11755ad65f76"} Oct 29 21:58:08.030546 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.030508 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:08.146529 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.146446 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\" (UniqueName: \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\") pod \"1cd83061-3276-4968-996f-efe710794e22\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " Oct 29 21:58:08.146529 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.146502 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\") pod \"1cd83061-3276-4968-996f-efe710794e22\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " Oct 29 21:58:08.146767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.146552 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\") pod \"1cd83061-3276-4968-996f-efe710794e22\" (UID: \"1cd83061-3276-4968-996f-efe710794e22\") " Oct 29 21:58:08.146854 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.146815 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca" pod "1cd83061-3276-4968-996f-efe710794e22" (UID: "1cd83061-3276-4968-996f-efe710794e22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:08.146953 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.146934 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd83061-3276-4968-996f-efe710794e22-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:08.148852 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.148828 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca" pod "1cd83061-3276-4968-996f-efe710794e22" (UID: "1cd83061-3276-4968-996f-efe710794e22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:08.148937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.148916 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb" pod "1cd83061-3276-4968-996f-efe710794e22" (UID: "1cd83061-3276-4968-996f-efe710794e22"). InnerVolumeSpecName "kube-api-access-ljvtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:08.248044 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.248019 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvtb\" (UniqueName: \"kubernetes.io/projected/1cd83061-3276-4968-996f-efe710794e22-kube-api-access-ljvtb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:08.248044 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.248044 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd83061-3276-4968-996f-efe710794e22-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:08.876877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.876850 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" Oct 29 21:58:08.876877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.876859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s" event={"ID":"1cd83061-3276-4968-996f-efe710794e22","Type":"ContainerDied","Data":"949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd"} Oct 29 21:58:08.877156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:08.876901 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949f5c2c73f1d24c38f28d3b5c5b4430ae753eaa38b1f2b45bd3847e49d894cd" Oct 29 21:58:09.737779 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.737744 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 21:58:09.738358 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.738341 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1cd83061-3276-4968-996f-efe710794e22" containerName="revoke-keys" Oct 29 21:58:09.738419 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.738359 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd83061-3276-4968-996f-efe710794e22" containerName="revoke-keys" Oct 29 21:58:09.738462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.738452 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd83061-3276-4968-996f-efe710794e22" containerName="revoke-keys" Oct 29 21:58:09.742146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.742129 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.776297 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.776274 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 21:58:09.865572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865541 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqsq\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865703 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865682 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865727 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865851 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865889 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.865967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.865951 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.866019 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.866006 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.866062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.866040 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967258 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967304 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967382 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967533 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967497 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqsq\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967586 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967550 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967586 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967578 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967697 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967647 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967697 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967690 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967715 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.967866 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.967812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.968126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.968099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.968251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.968127 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.968251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.968132 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.968251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.968175 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.970007 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.969987 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.970255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.970238 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.993032 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.992974 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:09.993153 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:09.993118 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqsq\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:10.055384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.055364 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:10.379385 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:10.379343 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037ce790_2dca_46d1_89a0_48fb94a8c693.slice/crio-e8b706fb33620d0ae838580b80a7ebecc519ed3965b3cafb8d4dd83bc292e4d8 WatchSource:0}: Error finding container e8b706fb33620d0ae838580b80a7ebecc519ed3965b3cafb8d4dd83bc292e4d8: Status 404 returned error can't find the container with id e8b706fb33620d0ae838580b80a7ebecc519ed3965b3cafb8d4dd83bc292e4d8 Oct 29 21:58:10.382549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.382507 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:10.382711 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.382595 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:10.382711 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.382633 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:10.386910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.386885 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 21:58:10.892592 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.892548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" event={"ID":"037ce790-2dca-46d1-89a0-48fb94a8c693","Type":"ContainerStarted","Data":"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5"} Oct 29 21:58:10.893030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.892608 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" event={"ID":"037ce790-2dca-46d1-89a0-48fb94a8c693","Type":"ContainerStarted","Data":"e8b706fb33620d0ae838580b80a7ebecc519ed3965b3cafb8d4dd83bc292e4d8"} Oct 29 21:58:10.945098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:10.943640 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podStartSLOduration=1.943608068 podStartE2EDuration="1.943608068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:10.943062508 +0000 UTC m=+952.684993181" watchObservedRunningTime="2026-10-29 21:58:10.943608068 +0000 UTC m=+952.685538741" Oct 29 21:58:11.055814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:11.055717 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:11.058068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:11.058038 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:11.058250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:11.058113 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:12.001936 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.001900 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 21:58:12.006487 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.006460 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.007496 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.007473 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 21:58:12.027089 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.027049 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"openshift-service-ca.crt\"" Oct 29 21:58:12.053098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.051373 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"distinct-w5-91520c-kserve-self-signed-certs\"" Oct 29 21:58:12.053098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.051436 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"kube-root-ca.crt\"" Oct 29 21:58:12.053098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.051519 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"default-dockercfg-g5d5n\"" Oct 29 21:58:12.055781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.055761 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:12.055940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.055916 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:12.093185 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093157 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.093285 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093190 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkgh\" (UniqueName: \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.093285 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093222 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.093371 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093349 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.093407 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093386 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.093448 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.093428 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.198099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.195432 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.198099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.195523 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.198099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.195587 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.211103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.203243 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.211103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.203312 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkgh\" (UniqueName: \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.211103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.203390 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.239502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.238958 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.239502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.239453 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.252144 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.248263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.252144 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.248415 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.255399 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.255370 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.284904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.284863 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkgh\" (UniqueName: \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\") pod \"distinct-w5-91520c-kserve-d77754bcb-6nw2x\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.324434 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.324403 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:12.666844 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.666796 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 21:58:12.906391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.906352 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" event={"ID":"6d0d9691-09aa-48be-87da-163df31eeef2","Type":"ContainerStarted","Data":"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe"} Oct 29 21:58:12.906543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.906401 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" event={"ID":"6d0d9691-09aa-48be-87da-163df31eeef2","Type":"ContainerStarted","Data":"28814f005afa6ea2d2df9b03a91770178d6609edbb23be7867e8afd0be8dac96"} Oct 29 21:58:12.957818 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:12.957733 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podStartSLOduration=1.957719408 podStartE2EDuration="1.957719408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:12.95342195 +0000 UTC m=+954.695352623" watchObservedRunningTime="2026-10-29 21:58:12.957719408 +0000 UTC m=+954.699650081" Oct 29 21:58:13.055802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.055768 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:13.056170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.055837 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:13.324984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.324955 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:13.855405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.855359 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k"] Oct 29 21:58:13.860774 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.860751 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:13.866821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.866785 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k"] Oct 29 21:58:13.891063 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.891043 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:58:13.891187 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.891152 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:58:13.891257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.891183 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:58:13.891257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.891194 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:58:13.923525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.923501 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:13.923672 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.923656 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbnb\" (UniqueName: \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:13.923721 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:13.923710 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.024641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.024604 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbnb\" (UniqueName: \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.024787 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.024675 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.024787 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.024741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.025421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.025397 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.027234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.027212 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.057112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.055865 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbnb\" (UniqueName: \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\") pod \"maas-api-revoke-sub-9e6a1bfb-kfd8k\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.057112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.056231 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:14.057112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.056277 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:14.180737 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.180673 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:14.507177 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.507147 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k"] Oct 29 21:58:14.507924 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:14.507900 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d7687c_36db_4e7a_bb3d_ba2f2648bba2.slice/crio-542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005 WatchSource:0}: Error finding container 542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005: Status 404 returned error can't find the container with id 542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005 Oct 29 21:58:14.922702 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:14.922669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" event={"ID":"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2","Type":"ContainerStarted","Data":"542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005"} Oct 29 21:58:15.057107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.056184 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:15.057107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.056241 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:15.556889 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.556840 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9"] Oct 29 21:58:15.561384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.561361 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.573062 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.572825 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9"] Oct 29 21:58:15.680251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.680226 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssh7w\" (UniqueName: \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.680365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.680274 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.680365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.680309 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.781281 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.781253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssh7w\" (UniqueName: \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.781437 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.781316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.781437 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.781361 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.781972 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.781947 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.783749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.783727 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.807650 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.807591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssh7w\" (UniqueName: \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\") pod \"maas-api-revoke-sub-4eeeb8e0-9cvl9\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.883665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.883635 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:15.958505 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.958214 4543 generic.go:356] "Generic (PLEG): container finished" podID="f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" containerID="efea16e9c1c39e484d82a3819c9e972206fcbb4be0c1e02fb97f816d31b71f58" exitCode=0 Oct 29 21:58:15.958505 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:15.958274 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" event={"ID":"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2","Type":"ContainerDied","Data":"efea16e9c1c39e484d82a3819c9e972206fcbb4be0c1e02fb97f816d31b71f58"} Oct 29 21:58:16.056929 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:16.056413 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:16.056929 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:16.056464 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:16.213545 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:16.213518 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9"] Oct 29 21:58:16.214218 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:16.214194 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b575ec0_348c_4a3a_a701_3fcb70959a1b.slice/crio-93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a WatchSource:0}: Error finding container 93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a: Status 404 returned error can't find the container with id 93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a Oct 29 21:58:16.966379 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:16.966343 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" event={"ID":"0b575ec0-348c-4a3a-a701-3fcb70959a1b","Type":"ContainerStarted","Data":"e928ec635483b3176b77f2cbcc49687439fee23f77f1759374c58b1bbbd3766a"} Oct 29 21:58:16.966554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:16.966391 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" event={"ID":"0b575ec0-348c-4a3a-a701-3fcb70959a1b","Type":"ContainerStarted","Data":"93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a"} Oct 29 21:58:17.014233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.014163 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" podStartSLOduration=2.01414933 podStartE2EDuration="2.01414933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:17.010748974 +0000 UTC m=+958.752679665" watchObservedRunningTime="2026-10-29 21:58:17.01414933 +0000 UTC m=+958.756080003" Oct 29 21:58:17.063097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.061733 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:58:17.063097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.061816 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 21:58:17.235143 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.224221 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:17.401772 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.401718 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\" (UniqueName: \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\") pod \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " Oct 29 21:58:17.402317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.402279 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\") pod \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " Oct 29 21:58:17.402481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.402376 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\") pod \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\" (UID: \"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2\") " Oct 29 21:58:17.402683 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.402657 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca" pod "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" (UID: "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:17.402984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.402960 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:17.404727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.404702 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb" pod "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" (UID: "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2"). InnerVolumeSpecName "kube-api-access-lzbnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:17.405010 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.404986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca" pod "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" (UID: "f3d7687c-36db-4e7a-bb3d-ba2f2648bba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:17.503905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.503835 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:17.503905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.503872 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbnb\" (UniqueName: \"kubernetes.io/projected/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2-kube-api-access-lzbnb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:17.975006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.974982 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" Oct 29 21:58:17.975170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.975016 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k" event={"ID":"f3d7687c-36db-4e7a-bb3d-ba2f2648bba2","Type":"ContainerDied","Data":"542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005"} Oct 29 21:58:17.975170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.975062 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="542741656ce5565586c2472732502e346d48db15e4adca93fed7d5b2ca48b005" Oct 29 21:58:17.976783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.976754 4543 generic.go:356] "Generic (PLEG): container finished" podID="0b575ec0-348c-4a3a-a701-3fcb70959a1b" containerID="e928ec635483b3176b77f2cbcc49687439fee23f77f1759374c58b1bbbd3766a" exitCode=0 Oct 29 21:58:17.976922 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:17.976798 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" event={"ID":"0b575ec0-348c-4a3a-a701-3fcb70959a1b","Type":"ContainerDied","Data":"e928ec635483b3176b77f2cbcc49687439fee23f77f1759374c58b1bbbd3766a"} Oct 29 21:58:18.060994 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:18.060969 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:18.061481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:18.061451 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:18.062295 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:18.062278 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 21:58:19.138726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.138702 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:19.218925 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.218894 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\" (UniqueName: \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\") pod \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " Oct 29 21:58:19.219065 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.219044 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\") pod \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " Oct 29 21:58:19.219150 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.219125 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\") pod \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\" (UID: \"0b575ec0-348c-4a3a-a701-3fcb70959a1b\") " Oct 29 21:58:19.219466 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.219437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca" pod "0b575ec0-348c-4a3a-a701-3fcb70959a1b" (UID: "0b575ec0-348c-4a3a-a701-3fcb70959a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:19.219626 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.219607 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:19.221049 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.221022 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w" pod "0b575ec0-348c-4a3a-a701-3fcb70959a1b" (UID: "0b575ec0-348c-4a3a-a701-3fcb70959a1b"). InnerVolumeSpecName "kube-api-access-ssh7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:19.221390 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.221372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca" pod "0b575ec0-348c-4a3a-a701-3fcb70959a1b" (UID: "0b575ec0-348c-4a3a-a701-3fcb70959a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:19.320775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.320720 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b575ec0-348c-4a3a-a701-3fcb70959a1b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:19.320775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.320746 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssh7w\" (UniqueName: \"kubernetes.io/projected/0b575ec0-348c-4a3a-a701-3fcb70959a1b-kube-api-access-ssh7w\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:19.990449 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.990407 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" Oct 29 21:58:19.990610 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.990476 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9" event={"ID":"0b575ec0-348c-4a3a-a701-3fcb70959a1b","Type":"ContainerDied","Data":"93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a"} Oct 29 21:58:19.990610 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:19.990529 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93628ddcfc000e75e3b3d7449ad19130ff7547f11ff3f1fd30f6486140158d9a" Oct 29 21:58:22.329106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:22.327029 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:22.373105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:22.371390 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:23.671434 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:23.671400 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:24.065770 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.065154 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.330149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.320152 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.395100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.394166 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.466356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.466309 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.545123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.544833 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.554982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.554950 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.927785 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:24.927564 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.075100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.074162 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.075100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.074619 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:58:25.353102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.286762 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.353102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.312216 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 29 21:58:25.367111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.363155 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.367111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.363587 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:58:25.446113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.416995 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.446113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.417386 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:58:25.446113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.428150 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:25.446113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.428196 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.680405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.680222 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.823188 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.716473 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.826642 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.755766 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.756145 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.755239 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.823349 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.827462 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:58:25.830102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.827527 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:58:25.913149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.895693 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:58:25.957881 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:25.924742 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:58:26.025946 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:26.024696 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:58:26.046983 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:26.028337 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:58:26.166669 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:26.166643 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:58:26.167280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:26.167260 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:58:27.962041 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:58:27.962005 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:58:31.439481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.439447 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 21:58:31.440225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440205 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" containerName="revoke-keys" Oct 29 21:58:31.440225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440226 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" containerName="revoke-keys" Oct 29 21:58:31.440391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440247 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0b575ec0-348c-4a3a-a701-3fcb70959a1b" containerName="revoke-keys" Oct 29 21:58:31.440391 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440254 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b575ec0-348c-4a3a-a701-3fcb70959a1b" containerName="revoke-keys" Oct 29 21:58:31.440497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440442 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" containerName="revoke-keys" Oct 29 21:58:31.440497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.440456 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b575ec0-348c-4a3a-a701-3fcb70959a1b" containerName="revoke-keys" Oct 29 21:58:31.445519 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.445499 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.469007 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.468982 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 21:58:31.478130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.478109 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-863f1b-openshift-default-dockercfg-5qt7r\"" Oct 29 21:58:31.571506 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571631 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571631 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571557 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571701 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571759 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571871 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571867 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbw2\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.571979 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.571954 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.572038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.572007 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673021 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673182 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673069 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbw2\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673182 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673140 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673189 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673269 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673327 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673299 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673337 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673596 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673533 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673622 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673824 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673881 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.673883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.674280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.673988 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.675842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.675821 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.676685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.676662 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.699117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.699039 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.699313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.699296 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbw2\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.758901 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.758880 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:31.800108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.800063 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd"] Oct 29 21:58:31.806132 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.806110 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.823984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.823730 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:58:31.823984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.823760 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:58:31.823984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.823760 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:58:31.823984 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.823838 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:58:31.834095 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.831368 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd"] Oct 29 21:58:31.875985 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.875959 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.876181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.876165 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.876247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.876231 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dnr\" (UniqueName: \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.977454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.977429 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.977598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.977565 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.977665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.977625 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dnr\" (UniqueName: \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.978120 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.978100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:31.979898 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:31.979880 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:32.003417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.003392 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dnr\" (UniqueName: \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\") pod \"maas-api-revoke-sub-f4da8fe8-wjnxd\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:32.086845 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.086716 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 21:58:32.089160 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:32.089132 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eb53b7_b336_4de1_95c4_035e94e3ab17.slice/crio-cb03c66e76eb9d3b249ff8a1f059b97abccb483a0132b0cbc66bad43ca35dd6a WatchSource:0}: Error finding container cb03c66e76eb9d3b249ff8a1f059b97abccb483a0132b0cbc66bad43ca35dd6a: Status 404 returned error can't find the container with id cb03c66e76eb9d3b249ff8a1f059b97abccb483a0132b0cbc66bad43ca35dd6a Oct 29 21:58:32.092329 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.092292 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:32.092430 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.092378 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:32.092487 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.092426 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:32.128991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.128973 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:32.453657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.453633 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd"] Oct 29 21:58:32.455012 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:32.454978 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34264ca8_86fe_43d0_a53e_f68ee5aa3b81.slice/crio-236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b WatchSource:0}: Error finding container 236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b: Status 404 returned error can't find the container with id 236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b Oct 29 21:58:32.981865 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.981827 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" event={"ID":"a6eb53b7-b336-4de1-95c4-035e94e3ab17","Type":"ContainerStarted","Data":"576001ca5abdc1550ddbeba7caf3dff34d24163e7c0054a9752cb007a6a44ef1"} Oct 29 21:58:32.982044 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.981913 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" event={"ID":"a6eb53b7-b336-4de1-95c4-035e94e3ab17","Type":"ContainerStarted","Data":"cb03c66e76eb9d3b249ff8a1f059b97abccb483a0132b0cbc66bad43ca35dd6a"} Oct 29 21:58:32.983525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.983496 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" event={"ID":"34264ca8-86fe-43d0-a53e-f68ee5aa3b81","Type":"ContainerStarted","Data":"7de752b1edf8884194bb78a93a538d79395f4e1a96b1ca41ede3c9ac1c695342"} Oct 29 21:58:32.983647 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:32.983552 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" event={"ID":"34264ca8-86fe-43d0-a53e-f68ee5aa3b81","Type":"ContainerStarted","Data":"236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b"} Oct 29 21:58:33.055101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.054498 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podStartSLOduration=2.054476547 podStartE2EDuration="2.054476547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:33.05138852 +0000 UTC m=+974.793319221" watchObservedRunningTime="2026-10-29 21:58:33.054476547 +0000 UTC m=+974.796407222" Oct 29 21:58:33.098924 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.098878 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" podStartSLOduration=2.098865253 podStartE2EDuration="2.098865253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:33.096835748 +0000 UTC m=+974.838766425" watchObservedRunningTime="2026-10-29 21:58:33.098865253 +0000 UTC m=+974.840795948" Oct 29 21:58:33.759407 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.759372 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:33.760867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.760841 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:33.761002 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.760899 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:33.991834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.991802 4543 generic.go:356] "Generic (PLEG): container finished" podID="34264ca8-86fe-43d0-a53e-f68ee5aa3b81" containerID="7de752b1edf8884194bb78a93a538d79395f4e1a96b1ca41ede3c9ac1c695342" exitCode=0 Oct 29 21:58:33.992013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:33.991884 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" event={"ID":"34264ca8-86fe-43d0-a53e-f68ee5aa3b81","Type":"ContainerDied","Data":"7de752b1edf8884194bb78a93a538d79395f4e1a96b1ca41ede3c9ac1c695342"} Oct 29 21:58:34.159454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.159380 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl"] Oct 29 21:58:34.163467 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.163450 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.182111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.182056 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl"] Oct 29 21:58:34.202614 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.202580 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.202730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.202651 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.202730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.202704 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trt2f\" (UniqueName: \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.304025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.303997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.304202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.304066 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.304202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.304129 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trt2f\" (UniqueName: \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.304727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.304704 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.306601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.306580 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.355747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.355721 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trt2f\" (UniqueName: \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\") pod \"maas-api-revoke-sub-2812bacc-5z6bl\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.479714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.479686 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:34.759727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.759659 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:34.759727 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.759710 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:34.806470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:34.806442 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl"] Oct 29 21:58:34.807686 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:34.807660 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0861bf56_903c_4b10_8f3c_f819b83d7dbf.slice/crio-efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6 WatchSource:0}: Error finding container efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6: Status 404 returned error can't find the container with id efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6 Oct 29 21:58:35.000160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.000132 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" event={"ID":"0861bf56-903c-4b10-8f3c-f819b83d7dbf","Type":"ContainerStarted","Data":"efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6"} Oct 29 21:58:35.201541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.201515 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:35.316823 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.316793 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\") pod \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " Oct 29 21:58:35.317011 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.316883 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\") pod \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " Oct 29 21:58:35.317011 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.316998 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\" (UniqueName: \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\") pod \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\" (UID: \"34264ca8-86fe-43d0-a53e-f68ee5aa3b81\") " Oct 29 21:58:35.321135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.317784 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca" pod "34264ca8-86fe-43d0-a53e-f68ee5aa3b81" (UID: "34264ca8-86fe-43d0-a53e-f68ee5aa3b81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:35.321135 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.318563 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:35.351270 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.350191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca" pod "34264ca8-86fe-43d0-a53e-f68ee5aa3b81" (UID: "34264ca8-86fe-43d0-a53e-f68ee5aa3b81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:35.355571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.355528 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr" pod "34264ca8-86fe-43d0-a53e-f68ee5aa3b81" (UID: "34264ca8-86fe-43d0-a53e-f68ee5aa3b81"). InnerVolumeSpecName "kube-api-access-f2dnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:35.419470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.419445 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:35.419470 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.419468 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dnr\" (UniqueName: \"kubernetes.io/projected/34264ca8-86fe-43d0-a53e-f68ee5aa3b81-kube-api-access-f2dnr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:35.760052 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.760016 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:35.760538 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:35.760109 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:36.009198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.009169 4543 generic.go:356] "Generic (PLEG): container finished" podID="0861bf56-903c-4b10-8f3c-f819b83d7dbf" containerID="6927efcdc104433b3f50c970d96e76c0c9dede3026077eb6c60f8715768b717e" exitCode=0 Oct 29 21:58:36.009338 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.009253 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" event={"ID":"0861bf56-903c-4b10-8f3c-f819b83d7dbf","Type":"ContainerDied","Data":"6927efcdc104433b3f50c970d96e76c0c9dede3026077eb6c60f8715768b717e"} Oct 29 21:58:36.010660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.010608 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" Oct 29 21:58:36.010660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.010634 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd" event={"ID":"34264ca8-86fe-43d0-a53e-f68ee5aa3b81","Type":"ContainerDied","Data":"236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b"} Oct 29 21:58:36.010800 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.010676 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236111f0eef92bc5cbde8aceaa6ff7f2d84da4913ed00a98d3741ce69d2d166b" Oct 29 21:58:36.760156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.760127 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:36.760527 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:36.760187 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:37.215411 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.215388 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:37.379165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.376598 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\") pod \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " Oct 29 21:58:37.379165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.376698 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\") pod \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " Oct 29 21:58:37.379165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.376777 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\" (UniqueName: \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\") pod \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\" (UID: \"0861bf56-903c-4b10-8f3c-f819b83d7dbf\") " Oct 29 21:58:37.379165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.378178 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca" pod "0861bf56-903c-4b10-8f3c-f819b83d7dbf" (UID: "0861bf56-903c-4b10-8f3c-f819b83d7dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:37.380036 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.379750 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f" pod "0861bf56-903c-4b10-8f3c-f819b83d7dbf" (UID: "0861bf56-903c-4b10-8f3c-f819b83d7dbf"). InnerVolumeSpecName "kube-api-access-trt2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:37.380505 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.380477 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca" pod "0861bf56-903c-4b10-8f3c-f819b83d7dbf" (UID: "0861bf56-903c-4b10-8f3c-f819b83d7dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:37.478284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.478247 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trt2f\" (UniqueName: \"kubernetes.io/projected/0861bf56-903c-4b10-8f3c-f819b83d7dbf-kube-api-access-trt2f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:37.478284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.478283 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:37.478284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.478294 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0861bf56-903c-4b10-8f3c-f819b83d7dbf-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:37.760131 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.760065 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:37.760288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:37.760151 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:38.025342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:38.025261 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" Oct 29 21:58:38.025342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:38.025292 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl" event={"ID":"0861bf56-903c-4b10-8f3c-f819b83d7dbf","Type":"ContainerDied","Data":"efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6"} Oct 29 21:58:38.025563 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:38.025344 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efb2711e46eef26a01758a13454b02c12f88510f77f603e8ca5b6add7900efb6" Oct 29 21:58:38.760024 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:38.759989 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:58:38.760234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:38.760048 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 21:58:39.763981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:39.763952 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:39.764465 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:39.764253 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:39.765177 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:39.765151 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 21:58:44.404720 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.404687 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn"] Oct 29 21:58:44.405438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405416 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0861bf56-903c-4b10-8f3c-f819b83d7dbf" containerName="revoke-keys" Oct 29 21:58:44.405539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405439 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0861bf56-903c-4b10-8f3c-f819b83d7dbf" containerName="revoke-keys" Oct 29 21:58:44.405539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405463 4543 cpu_manager.go:414] "containerMap: removing container" podUID="34264ca8-86fe-43d0-a53e-f68ee5aa3b81" containerName="revoke-keys" Oct 29 21:58:44.405539 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405470 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34264ca8-86fe-43d0-a53e-f68ee5aa3b81" containerName="revoke-keys" Oct 29 21:58:44.405679 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405665 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0861bf56-903c-4b10-8f3c-f819b83d7dbf" containerName="revoke-keys" Oct 29 21:58:44.405737 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.405680 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="34264ca8-86fe-43d0-a53e-f68ee5aa3b81" containerName="revoke-keys" Oct 29 21:58:44.409399 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.409377 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.427792 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.427768 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn"] Oct 29 21:58:44.437351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.437329 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:58:44.437760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.437742 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:58:44.437869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.437837 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:58:44.439760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.439740 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:58:44.555350 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.555311 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.555497 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.555402 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87pjr\" (UniqueName: \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.555555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.555493 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.656683 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.656598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.656683 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.656649 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87pjr\" (UniqueName: \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.656896 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.656721 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.657311 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.657283 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.664525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.664491 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.683329 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.683303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87pjr\" (UniqueName: \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\") pod \"maas-api-revoke-sub-d2b4a526-xw2rn\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:44.722627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:44.722590 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:45.037114 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:45.033652 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn"] Oct 29 21:58:45.038829 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:45.038800 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca31c8b2_b8ff_4430_b9c6_9df69b4e9854.slice/crio-b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34 WatchSource:0}: Error finding container b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34: Status 404 returned error can't find the container with id b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34 Oct 29 21:58:45.091655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:45.091631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" event={"ID":"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854","Type":"ContainerStarted","Data":"b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34"} Oct 29 21:58:46.099656 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:46.099625 4543 generic.go:356] "Generic (PLEG): container finished" podID="ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" containerID="e442d886d7a2a92f71c3cd2db41aa76c0f2e480b27645c443633ffeec03eede4" exitCode=0 Oct 29 21:58:46.099986 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:46.099706 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" event={"ID":"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854","Type":"ContainerDied","Data":"e442d886d7a2a92f71c3cd2db41aa76c0f2e480b27645c443633ffeec03eede4"} Oct 29 21:58:47.051847 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.051806 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 21:58:47.056179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.056155 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083346 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083310 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083360 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083453 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083497 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083532 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083566 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083605 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lv94\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.083934 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.083731 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.086253 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.086218 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 21:58:47.185110 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185045 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185132 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185175 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185216 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185265 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185383 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lv94\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185469 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185561 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185589 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185700 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.185995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.185950 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.186249 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.186223 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.186440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.186388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.188141 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.188120 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.192655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.192629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.216209 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.216186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lv94\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.216566 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.216538 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.287851 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.287828 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:47.372047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.371982 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:47.387691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.387671 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\") pod \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " Oct 29 21:58:47.387806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.387787 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\") pod \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " Oct 29 21:58:47.387883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.387867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\" (UniqueName: \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\") pod \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\" (UID: \"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854\") " Oct 29 21:58:47.388211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.388182 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca" pod "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" (UID: "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:58:47.388385 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.388370 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:47.390550 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.390527 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca" pod "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" (UID: "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:58:47.390657 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.390563 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr" pod "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" (UID: "ca31c8b2-b8ff-4430-b9c6-9df69b4e9854"). InnerVolumeSpecName "kube-api-access-87pjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:58:47.490029 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.489993 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:47.490182 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.490031 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87pjr\" (UniqueName: \"kubernetes.io/projected/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854-kube-api-access-87pjr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:58:47.697467 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.697439 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 21:58:47.697805 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:47.697783 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22af8b47_d881_4832_874f_12b90413feb0.slice/crio-ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e WatchSource:0}: Error finding container ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e: Status 404 returned error can't find the container with id ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e Oct 29 21:58:47.700305 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.700273 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:47.700385 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.700336 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:47.700421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:47.700391 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:48.123100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.122448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" event={"ID":"22af8b47-d881-4832-874f-12b90413feb0","Type":"ContainerStarted","Data":"3aefc17de225d12a960947c2cbf2b745f1136c49265fa3a3ae40ee76776f0868"} Oct 29 21:58:48.123100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.122515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" event={"ID":"22af8b47-d881-4832-874f-12b90413feb0","Type":"ContainerStarted","Data":"ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e"} Oct 29 21:58:48.124641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.124536 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" Oct 29 21:58:48.124641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.124578 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn" event={"ID":"ca31c8b2-b8ff-4430-b9c6-9df69b4e9854","Type":"ContainerDied","Data":"b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34"} Oct 29 21:58:48.124641 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.124633 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8df6be547db6c61dfe53d83d90bff0edfb1f5ef7f58d92baee33ef7bc49fe34" Oct 29 21:58:48.201137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.201059 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podStartSLOduration=1.201039002 podStartE2EDuration="1.201039002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:48.185508923 +0000 UTC m=+989.927439597" watchObservedRunningTime="2026-10-29 21:58:48.201039002 +0000 UTC m=+989.942969676" Oct 29 21:58:48.375045 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.373016 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:58:48.388097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.387310 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:48.388097 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:48.387373 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:49.373100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.372515 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:49.373100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.372602 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:49.745108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.730205 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 21:58:49.745108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.731411 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" containerName="revoke-keys" Oct 29 21:58:49.745108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.731435 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" containerName="revoke-keys" Oct 29 21:58:49.745108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.731579 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" containerName="revoke-keys" Oct 29 21:58:49.745108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.736150 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.757100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.754648 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 21:58:49.913100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.851916 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.913100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.852018 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.913100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.907240 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:49.913100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.908280 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913633 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913666 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913715 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913845 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.928106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.913888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbbq\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:49.958100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:49.935734 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.039158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.015837 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbbq\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.039158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.015979 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039730 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039762 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039837 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039904 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.039968 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.040320 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.040410 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.077130 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.040696 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.107601 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.079651 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.112222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.086178 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:58:50.116098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.086370 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.121463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.114530 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.173101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.171177 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.173101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.171579 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.187659 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.187200 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.209318 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.188458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.264976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.264587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.311146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.307858 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.345737 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.345688 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbbq\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.346543 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.346386 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.374540 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.374494 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:50.375122 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.375066 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:50.464405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.464368 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:50.955159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.924178 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.955159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.924987 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:58:50.955159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.951171 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.955159 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.951602 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:58:50.997123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.959158 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.997123 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:50.959561 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:58:51.187685 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.155666 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 21:58:51.204355 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.159145 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.159218 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.171801 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.171837 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.226329 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.226388 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.226425 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.226463 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:58:51.290572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.226496 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:58:51.371655 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.371616 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:51.371938 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.371893 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.407230 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.407193 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:51.417100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.411139 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:51.924822 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.924722 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.964151 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.958625 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.964499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:51.964461 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.154487 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.153278 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:58:52.233152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.219566 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.233152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.231543 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.233152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.231863 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 21:58:52.233152 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.232142 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 21:58:52.382776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.382744 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:52.392226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.392158 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:52.927111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:52.926227 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.033575 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.033517 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.167313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.049151 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.222104 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.220285 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.232324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.232190 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.256013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.233013 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:53.256587 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.256554 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:53.370976 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.370923 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:53.559646 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.253424 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:53.560019 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:58:53.386709 4543 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.001s" Oct 29 21:58:53.560328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.255308 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.560593 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.421552 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:53.560795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.560749 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:53.560949 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.453303 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.564222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.448437 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:58:53.710111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:53.710051 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" event={"ID":"7b6111b3-741b-43c8-9bee-4368f3817d0d","Type":"ContainerStarted","Data":"39da6e9a1938c252b1e08ecf022bf1ba9961c144535fbd7c349a656332c257eb"} Oct 29 21:58:54.026219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.020268 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 21:58:54.088504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.085457 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:54.100431 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.088755 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:58:54.125367 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.125301 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 21:58:54.220202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.219626 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:58:54.246099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.230391 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:58:54.249483 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.249461 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:58:54.266276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.266251 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:58:54.327401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.325619 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 21:58:54.327401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.325735 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:58:54.327401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.326692 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:58:54.361382 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.361295 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 21:58:54.374872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.367014 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.376730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.375550 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:54.376730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.375595 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:54.389440 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.389416 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569188 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569425 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569514 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569561 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569617 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569660 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569700 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569728 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.569782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzdsx\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.670627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.670546 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.670627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.670584 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.670627 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.670610 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.670903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.670654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.670903 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.670697 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671021 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671020 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671160 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671293 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671252 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdsx\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671300 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671678 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671731 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.671937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.671841 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.673781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.673757 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.674420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.674398 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.696701 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.696678 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.697047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.697029 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdsx\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.698921 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.698905 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:54.720795 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.720767 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" event={"ID":"7b6111b3-741b-43c8-9bee-4368f3817d0d","Type":"ContainerStarted","Data":"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a"} Oct 29 21:58:54.788607 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:54.787666 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podStartSLOduration=5.78765017 podStartE2EDuration="5.78765017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:54.779039116 +0000 UTC m=+996.520969790" watchObservedRunningTime="2026-10-29 21:58:54.78765017 +0000 UTC m=+996.529580902" Oct 29 21:58:55.084126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.084035 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 21:58:55.094514 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:55.094485 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e97e32_b52b_440b_b0a8_61979fb1e1c7.slice/crio-566e4b0a164a63684f94884e6d9aec87b3de021d8a279b57f150924cb3ab4a87 WatchSource:0}: Error finding container 566e4b0a164a63684f94884e6d9aec87b3de021d8a279b57f150924cb3ab4a87: Status 404 returned error can't find the container with id 566e4b0a164a63684f94884e6d9aec87b3de021d8a279b57f150924cb3ab4a87 Oct 29 21:58:55.098112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.097407 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:55.098112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.097507 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:55.098112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.097538 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:58:55.391156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.383566 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:55.391156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.383621 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:55.479117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.475145 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:58:55.494232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.494193 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:58:55.494405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.494249 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:58:55.790232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.790191 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" event={"ID":"a8e97e32-b52b-440b-b0a8-61979fb1e1c7","Type":"ContainerStarted","Data":"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863"} Oct 29 21:58:55.790415 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.790267 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" event={"ID":"a8e97e32-b52b-440b-b0a8-61979fb1e1c7","Type":"ContainerStarted","Data":"566e4b0a164a63684f94884e6d9aec87b3de021d8a279b57f150924cb3ab4a87"} Oct 29 21:58:55.866251 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:55.866185 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podStartSLOduration=2.866164924 podStartE2EDuration="2.866164924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:55.861226636 +0000 UTC m=+997.603157335" watchObservedRunningTime="2026-10-29 21:58:55.866164924 +0000 UTC m=+997.608095597" Oct 29 21:58:56.417096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.410710 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.417096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.410763 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:56.432111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.428212 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 21:58:56.472577 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.472537 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.472860 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.472832 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:58:56.508792 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.508752 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 21:58:56.509066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.508882 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.520834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.520834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.520834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.520834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.521218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.521218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.520873 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l9q\" (UniqueName: \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.549106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.548510 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"distinct-2-w5-91520c-kserve-self-signed-certs\"" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623237 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l9q\" (UniqueName: \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623420 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623515 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623565 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.623607 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.624460 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.625099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.625098 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.627155 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.627128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.681158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.681019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.715164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.703196 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.715164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.703280 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:58:56.715164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.703608 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:58:56.715164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.714876 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l9q\" (UniqueName: \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.715164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.715117 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\") pod \"distinct-2-w5-91520c-kserve-5766fc8cf-5bh69\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.825468 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.825202 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:56.921234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.921181 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2"] Oct 29 21:58:56.926913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.926887 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.004315 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.991307 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:58:57.004315 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.992184 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:58:57.004315 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:56.992346 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:58:57.024841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.023708 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzp8\" (UniqueName: \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.024841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.023767 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.024841 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.023892 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.045760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.045538 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:58:57.140102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.139235 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2"] Oct 29 21:58:57.180183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.140317 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzp8\" (UniqueName: \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.180183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.140367 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.180183 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.140473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.248105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.222230 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzp8\" (UniqueName: \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.248105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.225951 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.310149 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.300155 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b47e61-768z2\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.436102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.424188 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.436102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.424250 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:57.495880 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.484243 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.495880 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.484294 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:58:57.495880 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.485475 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:58:57.703531 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.703439 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.703531 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.703494 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:58:57.913651 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.913303 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 21:58:57.915700 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:58:57.915661 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebabb8b1_d6fb_436a_975a_1b93ca3c39c7.slice/crio-72919435179f6d08431bee30f28b9adc146bd451a976dee07d237da12bcf3522 WatchSource:0}: Error finding container 72919435179f6d08431bee30f28b9adc146bd451a976dee07d237da12bcf3522: Status 404 returned error can't find the container with id 72919435179f6d08431bee30f28b9adc146bd451a976dee07d237da12bcf3522 Oct 29 21:58:57.959098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:57.929940 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2"] Oct 29 21:58:58.209175 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:58:58.208122 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:58:58.389393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.388296 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.389393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.388357 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 21:58:58.482216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.482181 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.482782 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.482735 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:58:58.707690 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.707626 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.707690 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.707683 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:58:58.915571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.915494 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" event={"ID":"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7","Type":"ContainerStarted","Data":"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed"} Oct 29 21:58:58.915571 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.915542 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" event={"ID":"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7","Type":"ContainerStarted","Data":"72919435179f6d08431bee30f28b9adc146bd451a976dee07d237da12bcf3522"} Oct 29 21:58:58.917983 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.917956 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" event={"ID":"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8","Type":"ContainerStarted","Data":"ee31804f3f447b55028cb114d585025175b749ac99588cee00581bc35df583ef"} Oct 29 21:58:58.918067 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.918011 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" event={"ID":"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8","Type":"ContainerStarted","Data":"a324e8a513c3e3c727db44b145cf852bbe9bc4f88f9833ca7ef36a5466e5d791"} Oct 29 21:58:58.980907 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:58.980843 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podStartSLOduration=2.980823311 podStartE2EDuration="2.980823311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:58.970466086 +0000 UTC m=+1000.712396771" watchObservedRunningTime="2026-10-29 21:58:58.980823311 +0000 UTC m=+1000.722753985" Oct 29 21:58:59.020707 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.020660 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" podStartSLOduration=3.020642681 podStartE2EDuration="3.020642681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:59.019271492 +0000 UTC m=+1000.761202165" watchObservedRunningTime="2026-10-29 21:58:59.020642681 +0000 UTC m=+1000.762573357" Oct 29 21:58:59.477103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.477028 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:58:59.477278 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.477104 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:58:59.699806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.699714 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:58:59.699806 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.699778 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:58:59.825834 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.825802 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:58:59.960164 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.956690 4543 generic.go:356] "Generic (PLEG): container finished" podID="e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" containerID="ee31804f3f447b55028cb114d585025175b749ac99588cee00581bc35df583ef" exitCode=0 Oct 29 21:58:59.986100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:58:59.958711 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" event={"ID":"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8","Type":"ContainerDied","Data":"ee31804f3f447b55028cb114d585025175b749ac99588cee00581bc35df583ef"} Oct 29 21:59:00.433101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.432197 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:00.433101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.432284 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:00.483098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.466505 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:59:00.483098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.466577 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:59:00.699652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.699560 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:59:00.699652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.699634 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:59:00.987907 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.987880 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:59:00.988615 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.988556 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:59:00.990766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:00.990742 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 21:59:01.216954 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.216870 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:59:01.318783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.318690 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\") pod \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " Oct 29 21:59:01.318783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.318769 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\") pod \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " Oct 29 21:59:01.319030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.318812 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\" (UniqueName: \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\") pod \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\" (UID: \"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8\") " Oct 29 21:59:01.319199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.319173 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca" pod "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" (UID: "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:01.319757 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.319734 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:01.321154 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.321126 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca" pod "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" (UID: "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:01.321646 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.321621 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8" pod "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" (UID: "e649504e-5f89-4ca5-98d7-38a9e6e4b8a8"). InnerVolumeSpecName "kube-api-access-rzzp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:01.421420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.421390 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzzp8\" (UniqueName: \"kubernetes.io/projected/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-kube-api-access-rzzp8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:01.421420 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.421416 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:01.465652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.465617 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 29 21:59:01.465802 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.465666 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 29 21:59:01.699872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.699785 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:59:01.699872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.699854 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:59:01.984271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.984242 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" Oct 29 21:59:01.984469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.984273 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2" event={"ID":"e649504e-5f89-4ca5-98d7-38a9e6e4b8a8","Type":"ContainerDied","Data":"a324e8a513c3e3c727db44b145cf852bbe9bc4f88f9833ca7ef36a5466e5d791"} Oct 29 21:59:01.984469 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:01.984333 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a324e8a513c3e3c727db44b145cf852bbe9bc4f88f9833ca7ef36a5466e5d791" Oct 29 21:59:02.469885 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.469858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:59:02.470345 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.470333 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:59:02.471207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.471186 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 21:59:02.699754 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.699722 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 21:59:02.699982 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.699773 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 21:59:02.918100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.918011 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 21:59:02.919038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.919017 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" containerName="revoke-keys" Oct 29 21:59:02.919102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.919043 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" containerName="revoke-keys" Oct 29 21:59:02.919260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.919245 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" containerName="revoke-keys" Oct 29 21:59:02.923252 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.923227 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:02.941632 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.941607 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a116a2-dockercfg-7ht2n\"" Oct 29 21:59:02.942359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.942334 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a116a2\"" Oct 29 21:59:02.946013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:02.945986 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 21:59:03.011927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.011894 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 21:59:03.067100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.051586 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.068405 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.068366 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqvk\" (UniqueName: \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.068630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.068612 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.091100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.085844 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 21:59:03.171384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.170031 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.171384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.170167 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqvk\" (UniqueName: \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.171384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.170224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fbm\" (UniqueName: \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.171384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.170275 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.173423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.172557 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.197417 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.197395 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqvk\" (UniqueName: \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\") pod \"payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.252361 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.252333 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:03.271462 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.271436 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.271594 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.271535 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fbm\" (UniqueName: \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.272334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.272311 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.301582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.301550 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fbm\" (UniqueName: \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\") pod \"payload-processing-e2e-api-a-a116a2-865b999574-6j58r\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.464308 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.464280 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:03.571823 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.571786 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 21:59:03.703255 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.703226 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:59:03.703746 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.703725 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:59:03.704301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.704283 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 21:59:03.782848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:03.782823 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 21:59:03.782953 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:03.782868 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa957374_cdee_4db9_b4ac_69823f5c28c1.slice/crio-3c1552f5163622d8e8c6082dd7a8cc5f5ebba8e145b8bd38abe06b4e2e178ec7 WatchSource:0}: Error finding container 3c1552f5163622d8e8c6082dd7a8cc5f5ebba8e145b8bd38abe06b4e2e178ec7: Status 404 returned error can't find the container with id 3c1552f5163622d8e8c6082dd7a8cc5f5ebba8e145b8bd38abe06b4e2e178ec7 Oct 29 21:59:04.005210 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.005177 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" event={"ID":"b05722c4-32ff-465e-abcf-86c2d2f1a42f","Type":"ContainerStarted","Data":"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703"} Oct 29 21:59:04.005374 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.005229 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" event={"ID":"b05722c4-32ff-465e-abcf-86c2d2f1a42f","Type":"ContainerStarted","Data":"de155fb2ea6ed0ed3a5710fff1214196aab41fe51ea5c37fa6c1236385c827d2"} Oct 29 21:59:04.006122 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.006065 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:04.009643 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.009610 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" event={"ID":"fa957374-cdee-4db9-b4ac-69823f5c28c1","Type":"ContainerStarted","Data":"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415"} Oct 29 21:59:04.009760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.009664 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:04.009760 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.009691 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" event={"ID":"fa957374-cdee-4db9-b4ac-69823f5c28c1","Type":"ContainerStarted","Data":"3c1552f5163622d8e8c6082dd7a8cc5f5ebba8e145b8bd38abe06b4e2e178ec7"} Oct 29 21:59:04.053008 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.052918 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" podStartSLOduration=2.052905256 podStartE2EDuration="2.052905256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.049930269 +0000 UTC m=+1005.791860943" watchObservedRunningTime="2026-10-29 21:59:04.052905256 +0000 UTC m=+1005.794835928" Oct 29 21:59:04.095259 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.095222 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" podStartSLOduration=2.095210562 podStartE2EDuration="2.095210562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.093007847 +0000 UTC m=+1005.834938523" watchObservedRunningTime="2026-10-29 21:59:04.095210562 +0000 UTC m=+1005.837141235" Oct 29 21:59:04.806303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.806271 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk"] Oct 29 21:59:04.810431 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.810411 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.861435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.861409 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk"] Oct 29 21:59:04.870105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.870054 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:04.870218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.870152 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:04.870280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.870216 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:04.870371 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.870352 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:04.891831 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.891806 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.891940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.891844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.892068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.892051 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85t4\" (UniqueName: \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.992751 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.992729 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.992847 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.992756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.992897 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.992865 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85t4\" (UniqueName: \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.993421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.993398 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:04.995236 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:04.995216 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:05.019538 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:05.019514 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85t4\" (UniqueName: \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\") pod \"maas-api-revoke-sub-14c85271-cfctk\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:05.125261 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:05.125210 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:05.437904 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:05.437873 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk"] Oct 29 21:59:05.438275 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:05.438251 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb36c573_bd84_41d1_a531_f25467e895a0.slice/crio-73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0 WatchSource:0}: Error finding container 73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0: Status 404 returned error can't find the container with id 73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0 Oct 29 21:59:06.024094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:06.024012 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" event={"ID":"fb36c573-bd84-41d1-a531-f25467e895a0","Type":"ContainerStarted","Data":"f577b86be85c3323ccad63e9de3c1473ce6cf1fba7e08143a19160994471c8c8"} Oct 29 21:59:06.024094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:06.024070 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" event={"ID":"fb36c573-bd84-41d1-a531-f25467e895a0","Type":"ContainerStarted","Data":"73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0"} Oct 29 21:59:06.070536 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:06.070493 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" podStartSLOduration=2.070480291 podStartE2EDuration="2.070480291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:06.067630522 +0000 UTC m=+1007.809561192" watchObservedRunningTime="2026-10-29 21:59:06.070480291 +0000 UTC m=+1007.812410963" Oct 29 21:59:06.826403 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:06.826370 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:59:06.838784 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:06.838758 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:59:07.034271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:07.034238 4543 generic.go:356] "Generic (PLEG): container finished" podID="fb36c573-bd84-41d1-a531-f25467e895a0" containerID="f577b86be85c3323ccad63e9de3c1473ce6cf1fba7e08143a19160994471c8c8" exitCode=0 Oct 29 21:59:07.034773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:07.034326 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" event={"ID":"fb36c573-bd84-41d1-a531-f25467e895a0","Type":"ContainerDied","Data":"f577b86be85c3323ccad63e9de3c1473ce6cf1fba7e08143a19160994471c8c8"} Oct 29 21:59:07.045706 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:07.045681 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:59:08.201860 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.201837 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:08.333456 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.333388 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\") pod \"fb36c573-bd84-41d1-a531-f25467e895a0\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " Oct 29 21:59:08.333613 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.333589 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\") pod \"fb36c573-bd84-41d1-a531-f25467e895a0\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " Oct 29 21:59:08.333726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.333707 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\" (UniqueName: \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\") pod \"fb36c573-bd84-41d1-a531-f25467e895a0\" (UID: \"fb36c573-bd84-41d1-a531-f25467e895a0\") " Oct 29 21:59:08.333781 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.333765 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca" pod "fb36c573-bd84-41d1-a531-f25467e895a0" (UID: "fb36c573-bd84-41d1-a531-f25467e895a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:08.334141 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.334122 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:08.335729 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.335703 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca" pod "fb36c573-bd84-41d1-a531-f25467e895a0" (UID: "fb36c573-bd84-41d1-a531-f25467e895a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:08.335835 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.335814 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4" pod "fb36c573-bd84-41d1-a531-f25467e895a0" (UID: "fb36c573-bd84-41d1-a531-f25467e895a0"). InnerVolumeSpecName "kube-api-access-m85t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:08.434828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.434802 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb36c573-bd84-41d1-a531-f25467e895a0-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:08.434828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:08.434828 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85t4\" (UniqueName: \"kubernetes.io/projected/fb36c573-bd84-41d1-a531-f25467e895a0-kube-api-access-m85t4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:09.049977 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.049945 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" Oct 29 21:59:09.049977 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.049975 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk" event={"ID":"fb36c573-bd84-41d1-a531-f25467e895a0","Type":"ContainerDied","Data":"73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0"} Oct 29 21:59:09.050225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.050024 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73dff688b1a9afaa933fd63e5faae55393964ac992a487afdd1e5effcc92a8b0" Oct 29 21:59:09.898475 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.898447 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv"] Oct 29 21:59:09.899024 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.899010 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fb36c573-bd84-41d1-a531-f25467e895a0" containerName="revoke-keys" Oct 29 21:59:09.899069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.899025 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb36c573-bd84-41d1-a531-f25467e895a0" containerName="revoke-keys" Oct 29 21:59:09.899199 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.899187 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb36c573-bd84-41d1-a531-f25467e895a0" containerName="revoke-keys" Oct 29 21:59:09.902779 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.902763 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:09.922555 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.922533 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv"] Oct 29 21:59:09.925512 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.925487 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:09.931271 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.931251 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:09.931393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.931277 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:09.931393 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.931357 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:09.953404 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.953376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7kl\" (UniqueName: \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:09.953505 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.953474 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:09.953662 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:09.953643 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.054499 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.054468 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7kl\" (UniqueName: \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.054598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.054529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.054598 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.054589 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.055224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.055205 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.057179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.057159 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.080619 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.080593 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7kl\" (UniqueName: \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\") pod \"maas-api-revoke-sub-c27b7f41-vzvmv\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.217174 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.217153 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:10.528848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:10.528816 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv"] Oct 29 21:59:10.529382 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:10.529357 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8401a820_eded_4768_baca_88a723cfbb08.slice/crio-a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a WatchSource:0}: Error finding container a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a: Status 404 returned error can't find the container with id a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a Oct 29 21:59:11.065927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:11.065890 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" event={"ID":"8401a820-eded-4768-baca-88a723cfbb08","Type":"ContainerStarted","Data":"6efb499964fd0ed53ad85148cd975d8067f5cef24c89290f8626117b69848942"} Oct 29 21:59:11.066328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:11.065958 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" event={"ID":"8401a820-eded-4768-baca-88a723cfbb08","Type":"ContainerStarted","Data":"a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a"} Oct 29 21:59:11.113521 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:11.113440 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" podStartSLOduration=2.11342659 podStartE2EDuration="2.11342659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:11.109561153 +0000 UTC m=+1012.851491826" watchObservedRunningTime="2026-10-29 21:59:11.11342659 +0000 UTC m=+1012.855357262" Oct 29 21:59:12.074682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:12.074646 4543 generic.go:356] "Generic (PLEG): container finished" podID="8401a820-eded-4768-baca-88a723cfbb08" containerID="6efb499964fd0ed53ad85148cd975d8067f5cef24c89290f8626117b69848942" exitCode=0 Oct 29 21:59:12.075107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:12.074737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" event={"ID":"8401a820-eded-4768-baca-88a723cfbb08","Type":"ContainerDied","Data":"6efb499964fd0ed53ad85148cd975d8067f5cef24c89290f8626117b69848942"} Oct 29 21:59:13.230983 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.230959 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:13.297828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.297798 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\") pod \"8401a820-eded-4768-baca-88a723cfbb08\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " Oct 29 21:59:13.298004 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.297984 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\") pod \"8401a820-eded-4768-baca-88a723cfbb08\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " Oct 29 21:59:13.298088 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.298056 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\" (UniqueName: \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\") pod \"8401a820-eded-4768-baca-88a723cfbb08\" (UID: \"8401a820-eded-4768-baca-88a723cfbb08\") " Oct 29 21:59:13.298161 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.298132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca" pod "8401a820-eded-4768-baca-88a723cfbb08" (UID: "8401a820-eded-4768-baca-88a723cfbb08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:13.298701 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.298673 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8401a820-eded-4768-baca-88a723cfbb08-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:13.300204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.300178 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca" pod "8401a820-eded-4768-baca-88a723cfbb08" (UID: "8401a820-eded-4768-baca-88a723cfbb08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:13.300339 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.300316 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl" pod "8401a820-eded-4768-baca-88a723cfbb08" (UID: "8401a820-eded-4768-baca-88a723cfbb08"). InnerVolumeSpecName "kube-api-access-2z7kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:13.399873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.399825 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7kl\" (UniqueName: \"kubernetes.io/projected/8401a820-eded-4768-baca-88a723cfbb08-kube-api-access-2z7kl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:13.399873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:13.399849 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8401a820-eded-4768-baca-88a723cfbb08-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:14.091275 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:14.091249 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" Oct 29 21:59:14.091444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:14.091268 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv" event={"ID":"8401a820-eded-4768-baca-88a723cfbb08","Type":"ContainerDied","Data":"a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a"} Oct 29 21:59:14.091444 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:14.091323 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fd33209028532a20ab01e716dedb480031cab2c2029348e127280dd4530a6a" Oct 29 21:59:15.018070 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.018039 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 21:59:15.155603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.155571 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv"] Oct 29 21:59:15.156288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.156267 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8401a820-eded-4768-baca-88a723cfbb08" containerName="revoke-keys" Oct 29 21:59:15.156288 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.156288 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8401a820-eded-4768-baca-88a723cfbb08" containerName="revoke-keys" Oct 29 21:59:15.156478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.156465 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8401a820-eded-4768-baca-88a723cfbb08" containerName="revoke-keys" Oct 29 21:59:15.160125 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.160103 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.181818 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.181790 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:15.181999 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.181831 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:15.187280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.187257 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv"] Oct 29 21:59:15.190775 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.190750 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:15.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.190879 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:15.218365 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.218331 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.218595 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.218570 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94gr\" (UniqueName: \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.218676 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.218650 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.320426 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.320366 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.320533 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.320476 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94gr\" (UniqueName: \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.320533 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.320517 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.321175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.321151 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.322870 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.322853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.346534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.346515 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94gr\" (UniqueName: \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\") pod \"maas-api-revoke-sub-e1ad6b62-kthdv\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.487033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.487012 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:15.793095 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:15.793053 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv"] Oct 29 21:59:15.793561 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:15.793534 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cced06e_25ae_439b_94b5_28d6f93d8c15.slice/crio-4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3 WatchSource:0}: Error finding container 4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3: Status 404 returned error can't find the container with id 4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3 Oct 29 21:59:16.027103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:16.027061 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 21:59:16.113701 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:16.113664 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" event={"ID":"6cced06e-25ae-439b-94b5-28d6f93d8c15","Type":"ContainerStarted","Data":"e82e3a7a5017cb2165db0ee1a8ab6d98e47f45865211f0d40bec38dc0ba7c2be"} Oct 29 21:59:16.113884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:16.113717 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" event={"ID":"6cced06e-25ae-439b-94b5-28d6f93d8c15","Type":"ContainerStarted","Data":"4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3"} Oct 29 21:59:16.176790 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:16.176740 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" podStartSLOduration=1.176725852 podStartE2EDuration="1.176725852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:16.172553376 +0000 UTC m=+1017.914484061" watchObservedRunningTime="2026-10-29 21:59:16.176725852 +0000 UTC m=+1017.918656521" Oct 29 21:59:17.016534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.016498 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 21:59:17.024967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.024926 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.057111 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.057063 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 21:59:17.125435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.125392 4543 generic.go:356] "Generic (PLEG): container finished" podID="6cced06e-25ae-439b-94b5-28d6f93d8c15" containerID="e82e3a7a5017cb2165db0ee1a8ab6d98e47f45865211f0d40bec38dc0ba7c2be" exitCode=0 Oct 29 21:59:17.125435 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.125425 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" event={"ID":"6cced06e-25ae-439b-94b5-28d6f93d8c15","Type":"ContainerDied","Data":"e82e3a7a5017cb2165db0ee1a8ab6d98e47f45865211f0d40bec38dc0ba7c2be"} Oct 29 21:59:17.157607 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157578 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.157732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157614 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.157732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.157732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157714 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.157854 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157809 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94jpz\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.157939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157924 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.158006 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.157993 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.158057 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.158045 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.158113 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.158087 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.261704 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.261868 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94jpz\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262006 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262141 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262178 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262264 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262293 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262375 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.264485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.262842 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.265259 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.265108 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.265911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.265879 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.266117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.266049 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.267698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.267629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.267902 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.267878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.268222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.268203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.291413 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.291384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94jpz\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.291955 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.291935 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.343698 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.343679 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:17.673324 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.673295 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 21:59:17.674525 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:17.674498 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c01e7d_527e_4ee6_9298_3a7d7bfb9078.slice/crio-b29a3e0983404adc87c9ae91e286ccadee3a8a61edc7e78bfce91d75c3813cea WatchSource:0}: Error finding container b29a3e0983404adc87c9ae91e286ccadee3a8a61edc7e78bfce91d75c3813cea: Status 404 returned error can't find the container with id b29a3e0983404adc87c9ae91e286ccadee3a8a61edc7e78bfce91d75c3813cea Oct 29 21:59:17.677436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.677397 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:17.677544 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.677481 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:17.677544 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:17.677530 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:18.135714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.135679 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" event={"ID":"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078","Type":"ContainerStarted","Data":"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec"} Oct 29 21:59:18.136233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.135731 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" event={"ID":"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078","Type":"ContainerStarted","Data":"b29a3e0983404adc87c9ae91e286ccadee3a8a61edc7e78bfce91d75c3813cea"} Oct 29 21:59:18.176364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.176277 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 21:59:18.181886 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.181862 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.196665 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.196617 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podStartSLOduration=2.196604393 podStartE2EDuration="2.196604393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:18.188649833 +0000 UTC m=+1019.930580551" watchObservedRunningTime="2026-10-29 21:59:18.196604393 +0000 UTC m=+1019.938535066" Oct 29 21:59:18.208433 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.208395 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 21:59:18.276022 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.275977 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276043 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48gj\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276190 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276227 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276381 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276262 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276446 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276396 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276568 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276550 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276638 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276622 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.276691 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.276681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.292520 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.292500 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:18.344669 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.344640 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:18.346227 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.346201 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:18.346360 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.346330 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:18.378186 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378160 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\") pod \"6cced06e-25ae-439b-94b5-28d6f93d8c15\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " Oct 29 21:59:18.378353 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378336 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\" (UniqueName: \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\") pod \"6cced06e-25ae-439b-94b5-28d6f93d8c15\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " Oct 29 21:59:18.378421 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378393 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\") pod \"6cced06e-25ae-439b-94b5-28d6f93d8c15\" (UID: \"6cced06e-25ae-439b-94b5-28d6f93d8c15\") " Oct 29 21:59:18.378654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378632 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378725 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378698 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378784 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378746 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378839 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378812 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378891 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48gj\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378961 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378909 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.378961 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378933 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.379068 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.378963 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.379140 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.379091 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.379531 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.379504 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca" pod "6cced06e-25ae-439b-94b5-28d6f93d8c15" (UID: "6cced06e-25ae-439b-94b5-28d6f93d8c15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:18.380066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.380036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.380783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.380481 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.380783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.380533 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.380783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.380744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.381276 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.381255 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.382476 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.382456 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.383823 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.383796 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr" pod "6cced06e-25ae-439b-94b5-28d6f93d8c15" (UID: "6cced06e-25ae-439b-94b5-28d6f93d8c15"). InnerVolumeSpecName "kube-api-access-f94gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:18.384007 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.383987 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca" pod "6cced06e-25ae-439b-94b5-28d6f93d8c15" (UID: "6cced06e-25ae-439b-94b5-28d6f93d8c15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:18.384749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.384727 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.405297 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.405269 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48gj\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.405479 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.405464 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.480283 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.480257 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:18.480423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.480296 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94gr\" (UniqueName: \"kubernetes.io/projected/6cced06e-25ae-439b-94b5-28d6f93d8c15-kube-api-access-f94gr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:18.480423 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.480320 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cced06e-25ae-439b-94b5-28d6f93d8c15-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:18.506390 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.506343 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:18.871812 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.871784 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 21:59:18.872210 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:18.872186 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea783f1_8102_4110_a4d2_be7a6e509183.slice/crio-fcdf6b6f53b2143917d34cd79d7109e285e0f789c2499b1416f88c549bcdd0dd WatchSource:0}: Error finding container fcdf6b6f53b2143917d34cd79d7109e285e0f789c2499b1416f88c549bcdd0dd: Status 404 returned error can't find the container with id fcdf6b6f53b2143917d34cd79d7109e285e0f789c2499b1416f88c549bcdd0dd Oct 29 21:59:18.882277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.882228 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:18.882522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.882495 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:18.882745 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:18.882723 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:19.162892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.162800 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" event={"ID":"fea783f1-8102-4110-a4d2-be7a6e509183","Type":"ContainerStarted","Data":"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d"} Oct 29 21:59:19.162892 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.162891 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" event={"ID":"fea783f1-8102-4110-a4d2-be7a6e509183","Type":"ContainerStarted","Data":"fcdf6b6f53b2143917d34cd79d7109e285e0f789c2499b1416f88c549bcdd0dd"} Oct 29 21:59:19.164494 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.164472 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" Oct 29 21:59:19.164625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.164500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv" event={"ID":"6cced06e-25ae-439b-94b5-28d6f93d8c15","Type":"ContainerDied","Data":"4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3"} Oct 29 21:59:19.164625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.164553 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be9f45c4e6fab708a0d011cd6cae5f0bbfd99845515d2201b14538bf46377a3" Oct 29 21:59:19.219057 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.219015 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podStartSLOduration=1.219001883 podStartE2EDuration="1.219001883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:19.216212975 +0000 UTC m=+1020.958143650" watchObservedRunningTime="2026-10-29 21:59:19.219001883 +0000 UTC m=+1020.960932555" Oct 29 21:59:19.346101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.345560 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:19.346101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.345631 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:19.507202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.507176 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:19.508640 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.508607 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:19.508778 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.508655 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:19.891158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.890402 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 21:59:19.891399 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.891376 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6cced06e-25ae-439b-94b5-28d6f93d8c15" containerName="revoke-keys" Oct 29 21:59:19.891472 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.891401 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cced06e-25ae-439b-94b5-28d6f93d8c15" containerName="revoke-keys" Oct 29 21:59:19.891643 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.891628 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cced06e-25ae-439b-94b5-28d6f93d8c15" containerName="revoke-keys" Oct 29 21:59:19.904103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.903089 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 21:59:19.904103 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:19.903199 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003179 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003228 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003517 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003391 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003517 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003450 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003544 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003642 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003591 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003750 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003656 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003750 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003685 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.003872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.003802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpc9r\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.104860 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.104821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.104874 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.104984 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105091 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105136 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105203 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105250 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105235 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105311 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpc9r\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.105798 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.105769 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.108117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.106341 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.108117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.107001 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.108117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.107340 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.109272 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.108415 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.110246 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.110224 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.191261 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.188862 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.192939 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.192916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpc9r\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.221402 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.221198 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:20.310049 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.310017 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 21:59:20.316734 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.316705 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.340489 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.338716 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8867e7d-dockercfg-c96bh\"" Oct 29 21:59:20.376277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.353273 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8867e7d\"" Oct 29 21:59:20.376277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.354238 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.376277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.354285 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:20.411380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.411350 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.411560 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.411495 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4ks\" (UniqueName: \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.413175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.413151 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 21:59:20.531235 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.531200 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.581031 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.580990 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:20.581304 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.559593 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.584508 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.558440 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.655227 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.655190 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4ks\" (UniqueName: \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.703212 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.701152 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt"] Oct 29 21:59:20.727988 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.719019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4ks\" (UniqueName: \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.727988 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.719570 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:20.781686 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.781606 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.793151 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.792047 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt"] Oct 29 21:59:20.804452 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.804407 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:20.804770 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.804726 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:20.804915 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.804873 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:20.812718 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.812700 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wwn\" (UniqueName: \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.813033 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.813019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.813305 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.813293 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.874066 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.874020 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:20.892323 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:20.892245 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39396390_b94e_42fa_ae91_e837573e4009.slice/crio-d08e886431a0dbfb5fdbbf17f42c06dd789456bc4b8bd9ed550e4ed06987156f WatchSource:0}: Error finding container d08e886431a0dbfb5fdbbf17f42c06dd789456bc4b8bd9ed550e4ed06987156f: Status 404 returned error can't find the container with id d08e886431a0dbfb5fdbbf17f42c06dd789456bc4b8bd9ed550e4ed06987156f Oct 29 21:59:20.895247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.894814 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:20.895247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.894906 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:20.895247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.894949 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:20.902842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.902820 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 21:59:20.914713 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.914686 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.915609 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.915585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.915814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.915792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wwn\" (UniqueName: \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.916523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.916502 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.918847 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.918826 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:20.984993 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:20.984939 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wwn\" (UniqueName: \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\") pod \"maas-api-revoke-sub-4722f60b-2hdqt\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:21.186940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.185027 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:21.186940 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.185662 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 21:59:21.205247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.205221 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" event={"ID":"39396390-b94e-42fa-ae91-e837573e4009","Type":"ContainerStarted","Data":"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca"} Oct 29 21:59:21.205636 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.205604 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" event={"ID":"39396390-b94e-42fa-ae91-e837573e4009","Type":"ContainerStarted","Data":"d08e886431a0dbfb5fdbbf17f42c06dd789456bc4b8bd9ed550e4ed06987156f"} Oct 29 21:59:21.247800 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.224195 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:21.281098 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.259844 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podStartSLOduration=2.2598243670000002 podStartE2EDuration="2.259824367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:21.257490219 +0000 UTC m=+1022.999420904" watchObservedRunningTime="2026-10-29 21:59:21.259824367 +0000 UTC m=+1023.001755044" Oct 29 21:59:21.354349 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.345864 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.354349 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.345913 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:21.507954 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.507899 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.508117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.507967 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:21.586706 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:21.586665 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c86ac86_6cfe_423a_a6dd_f890de2fb48c.slice/crio-d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6 WatchSource:0}: Error finding container d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6: Status 404 returned error can't find the container with id d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6 Oct 29 21:59:21.592970 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:21.592934 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt"] Oct 29 21:59:22.222916 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.222881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" event={"ID":"7c86ac86-6cfe-423a-a6dd-f890de2fb48c","Type":"ContainerStarted","Data":"d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6"} Oct 29 21:59:22.224258 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.224233 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.224392 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.224273 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:22.226854 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.226827 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" event={"ID":"214d1999-43cc-48c3-aee6-b9210eaea845","Type":"ContainerStarted","Data":"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7"} Oct 29 21:59:22.227005 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.226993 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" event={"ID":"214d1999-43cc-48c3-aee6-b9210eaea845","Type":"ContainerStarted","Data":"e692a9efb49e82e962f478451fe92862a1dbc127e207990084196871b3970787"} Oct 29 21:59:22.227147 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.227136 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:22.296132 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.278400 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" podStartSLOduration=2.278380956 podStartE2EDuration="2.278380956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:22.277537065 +0000 UTC m=+1024.019467739" watchObservedRunningTime="2026-10-29 21:59:22.278380956 +0000 UTC m=+1024.020311630" Oct 29 21:59:22.347458 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.347427 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.347912 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.347877 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:22.516100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.507524 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.516100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:22.507593 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:23.221989 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.221950 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.222314 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.222283 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:23.295196 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.295155 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c86ac86-6cfe-423a-a6dd-f890de2fb48c" containerID="2a160da2fb87539be964e5caeb7096c074c267d15539be8b77f2a7087e89c52d" exitCode=0 Oct 29 21:59:23.297117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.296061 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" event={"ID":"7c86ac86-6cfe-423a-a6dd-f890de2fb48c","Type":"ContainerDied","Data":"2a160da2fb87539be964e5caeb7096c074c267d15539be8b77f2a7087e89c52d"} Oct 29 21:59:23.365265 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.365219 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.365565 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.365517 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:23.511448 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.511346 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.511716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:23.511663 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:24.222815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.222769 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.223003 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.222834 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:24.346047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.345607 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.346047 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.345666 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 29 21:59:24.512234 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.512140 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.512463 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:24.512436 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:25.229226 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.222257 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:25.229504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.229479 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.229684 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.229656 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:25.324820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.324724 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\") pod \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " Oct 29 21:59:25.325165 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.325143 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\") pod \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " Oct 29 21:59:25.325367 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.325353 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\" (UniqueName: \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\") pod \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\" (UID: \"7c86ac86-6cfe-423a-a6dd-f890de2fb48c\") " Oct 29 21:59:25.326502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.326465 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca" pod "7c86ac86-6cfe-423a-a6dd-f890de2fb48c" (UID: "7c86ac86-6cfe-423a-a6dd-f890de2fb48c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:25.405100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.403520 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" event={"ID":"7c86ac86-6cfe-423a-a6dd-f890de2fb48c","Type":"ContainerDied","Data":"d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6"} Oct 29 21:59:25.405100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.403574 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2abf9ce1699602152d96642c54d9786c4ef68c0317c39830758a246b290e9b6" Oct 29 21:59:25.405100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.403634 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt" Oct 29 21:59:25.440661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.437428 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:25.485827 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.477430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn" pod "7c86ac86-6cfe-423a-a6dd-f890de2fb48c" (UID: "7c86ac86-6cfe-423a-a6dd-f890de2fb48c"). InnerVolumeSpecName "kube-api-access-67wwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:25.498636 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.497254 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca" pod "7c86ac86-6cfe-423a-a6dd-f890de2fb48c" (UID: "7c86ac86-6cfe-423a-a6dd-f890de2fb48c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:25.530106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.527391 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.530106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.527457 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:25.550804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.550684 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:25.551032 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:25.551011 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wwn\" (UniqueName: \"kubernetes.io/projected/7c86ac86-6cfe-423a-a6dd-f890de2fb48c-kube-api-access-67wwn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:26.375158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.374175 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:26.375158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.374248 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.446108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.432363 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.446108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.432422 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:26.528057 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.527647 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.528057 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.527707 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 29 21:59:26.868796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:26.868746 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.114247 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.114195 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.228910 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.228875 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:27.229290 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.229240 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:27.285101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.283539 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:59:27.311436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.311393 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.340819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.340776 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.341378 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.341349 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 21:59:27.342552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.342527 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.387216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.387179 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:27.387542 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.387510 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.444190 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.444149 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.746308 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.746231 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:27.746969 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.746932 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.884359 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:27.884301 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": context deadline exceeded" Oct 29 21:59:28.142158 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.122534 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 21:59:28.246107 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.189984 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.265260 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.265205 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.353108 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.318194 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 21:59:28.406156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.318233 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 29 21:59:28.406156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.326491 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:59:28.406156 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.401279 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:59:28.450140 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.191546 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 21:59:28.770157 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.399181 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 21:59:28.860179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.399359 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 21:59:28.886414 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.351313 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:28.886848 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.886688 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:28.953101 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.454234 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 21:59:28.976401 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.454313 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.976652 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.398910 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 21:59:28.976872 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.583666 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:28.980194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.980154 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.990257 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:28.704347 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:29.015202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.015157 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.253100 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:59:29.227923 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:59:29.265799 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.253382 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.268446 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.268411 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.320325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.320276 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.441513 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.404730 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.441738 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.405362 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:29.441884 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.441857 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:29.520695 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:59:29.520547 4543 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.066s" Oct 29 21:59:29.532193 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.532168 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:59:29.532436 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.532422 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:59:29.532617 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.532605 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:59:29.532747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.532735 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:59:29.578958 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.578933 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:59:29.580277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.580258 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:59:29.580441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.580431 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 21:59:29.580558 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.580549 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 21:59:29.636109 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.632854 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 21:59:29.789426 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.789328 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.791105 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.791057 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:59:29.815631 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.815580 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 21:59:29.865325 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.865288 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.993102 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:29.987200 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:30.131287 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.131167 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 21:59:30.142094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.138175 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:30.142094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.138230 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:30.142094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.139057 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:30.142094 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.139277 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:30.144127 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.144049 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 21:59:30.149883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.148340 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:30.149883 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.149346 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:30.158674 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.158615 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 21:59:30.197127 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.185385 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 21:59:30.222733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.222698 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 29 21:59:30.225931 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.225884 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 29 21:59:30.265136 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.264383 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:30.296683 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.296503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 21:59:30.318911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.318887 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 21:59:30.341037 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:30.340936 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 21:59:31.179272 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.179229 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:31.179744 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.179299 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:31.501415 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.501386 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:31.502018 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.501992 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:31.502730 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.502707 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 21:59:31.647216 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.647185 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:31.648204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.648173 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:31.649263 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:31.649233 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 21:59:32.743905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.743868 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs"] Oct 29 21:59:32.744554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.744536 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c86ac86-6cfe-423a-a6dd-f890de2fb48c" containerName="revoke-keys" Oct 29 21:59:32.744554 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.744552 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c86ac86-6cfe-423a-a6dd-f890de2fb48c" containerName="revoke-keys" Oct 29 21:59:32.744726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.744642 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c86ac86-6cfe-423a-a6dd-f890de2fb48c" containerName="revoke-keys" Oct 29 21:59:32.749688 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.749665 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.774986 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.774949 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs"] Oct 29 21:59:32.778534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.778513 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:32.778675 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.778515 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:32.778749 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.778573 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:32.786315 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.786292 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:32.811117 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.811068 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.811455 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.811422 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.811625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.811602 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62lj\" (UniqueName: \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.912623 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.912594 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62lj\" (UniqueName: \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.912796 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.912697 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.912867 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.912798 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.913486 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.913459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.915280 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.915261 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:32.940351 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:32.940282 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62lj\" (UniqueName: \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\") pod \"maas-api-revoke-sub-a5f49ac2-4fvbs\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:33.076674 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.076605 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:33.460219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.460135 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 21:59:33.475534 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.475506 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.480198 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:33.480151 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb804c076_6c83_47d5_8fc1_0b259492dee4.slice/crio-0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252 WatchSource:0}: Error finding container 0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252: Status 404 returned error can't find the container with id 0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252 Oct 29 21:59:33.491181 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.488898 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 21:59:33.495347 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.495319 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs"] Oct 29 21:59:33.583010 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.582989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583200 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583327 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583304 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583385 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583357 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj25g\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583442 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583423 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583493 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583481 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583649 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583634 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.583723 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.583707 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685224 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685200 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685308 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685252 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685308 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685380 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685327 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685438 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685664 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685648 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685741 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685712 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685719 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685809 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj25g\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685842 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685942 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.685981 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685967 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.686031 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.685981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.687968 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.687947 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.688089 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.688057 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.711059 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.711040 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.711328 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.711284 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj25g\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:33.812174 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:33.812153 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:34.140600 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:34.140569 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a3bfb9_9b82_47c7_8092_525e92bfca3f.slice/crio-efd8e197760deec6d539942eaf0884bb335fca934e3bd49370f4ac0d6bf00d7a WatchSource:0}: Error finding container efd8e197760deec6d539942eaf0884bb335fca934e3bd49370f4ac0d6bf00d7a: Status 404 returned error can't find the container with id efd8e197760deec6d539942eaf0884bb335fca934e3bd49370f4ac0d6bf00d7a Oct 29 21:59:34.140747 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.140726 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 21:59:34.143454 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.143412 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:34.143613 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.143499 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:34.143613 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.143545 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:34.315862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.315832 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 21:59:34.422267 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.422194 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" event={"ID":"f7a3bfb9-9b82-47c7-8092-525e92bfca3f","Type":"ContainerStarted","Data":"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656"} Oct 29 21:59:34.422267 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.422258 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" event={"ID":"f7a3bfb9-9b82-47c7-8092-525e92bfca3f","Type":"ContainerStarted","Data":"efd8e197760deec6d539942eaf0884bb335fca934e3bd49370f4ac0d6bf00d7a"} Oct 29 21:59:34.423941 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.423918 4543 generic.go:356] "Generic (PLEG): container finished" podID="b804c076-6c83-47d5-8fc1-0b259492dee4" containerID="4b681318313d008ab288ae04eea5cc72aae2869db6b68f78e32cae6b9a3567a7" exitCode=0 Oct 29 21:59:34.424056 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.423997 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" event={"ID":"b804c076-6c83-47d5-8fc1-0b259492dee4","Type":"ContainerDied","Data":"4b681318313d008ab288ae04eea5cc72aae2869db6b68f78e32cae6b9a3567a7"} Oct 29 21:59:34.424056 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.424038 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" event={"ID":"b804c076-6c83-47d5-8fc1-0b259492dee4","Type":"ContainerStarted","Data":"0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252"} Oct 29 21:59:34.484061 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.484013 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podStartSLOduration=1.483998726 podStartE2EDuration="1.483998726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:34.479032843 +0000 UTC m=+1036.220963528" watchObservedRunningTime="2026-10-29 21:59:34.483998726 +0000 UTC m=+1036.225929399" Oct 29 21:59:34.812900 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:34.812866 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:35.665357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.665322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:35.716438 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.716403 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\") pod \"b804c076-6c83-47d5-8fc1-0b259492dee4\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " Oct 29 21:59:35.716625 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.716571 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\" (UniqueName: \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\") pod \"b804c076-6c83-47d5-8fc1-0b259492dee4\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " Oct 29 21:59:35.716695 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.716672 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\") pod \"b804c076-6c83-47d5-8fc1-0b259492dee4\" (UID: \"b804c076-6c83-47d5-8fc1-0b259492dee4\") " Oct 29 21:59:35.717673 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.717646 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca" pod "b804c076-6c83-47d5-8fc1-0b259492dee4" (UID: "b804c076-6c83-47d5-8fc1-0b259492dee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:35.719808 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.719780 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca" pod "b804c076-6c83-47d5-8fc1-0b259492dee4" (UID: "b804c076-6c83-47d5-8fc1-0b259492dee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:35.725120 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.720119 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj" pod "b804c076-6c83-47d5-8fc1-0b259492dee4" (UID: "b804c076-6c83-47d5-8fc1-0b259492dee4"). InnerVolumeSpecName "kube-api-access-f62lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:35.814138 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.814108 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:35.814489 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.814150 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:35.818326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.818290 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62lj\" (UniqueName: \"kubernetes.io/projected/b804c076-6c83-47d5-8fc1-0b259492dee4-kube-api-access-f62lj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:35.818326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.818323 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:35.818326 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:35.818333 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b804c076-6c83-47d5-8fc1-0b259492dee4-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:36.463364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:36.463332 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" Oct 29 21:59:36.463597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:36.463367 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs" event={"ID":"b804c076-6c83-47d5-8fc1-0b259492dee4","Type":"ContainerDied","Data":"0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252"} Oct 29 21:59:36.463597 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:36.463419 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd9f1fe5ab5814a5c705787bed91e745c72200c4afd883aa5c6c60e80ebc252" Oct 29 21:59:36.812737 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:36.812707 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:36.812891 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:36.812756 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:37.813220 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:37.813188 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:37.813602 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:37.813243 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:38.813194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:38.813156 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:38.813364 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:38.813212 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:39.813254 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:39.813221 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:39.813429 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:39.813273 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:40.812676 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:40.812647 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 29 21:59:40.812864 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:40.812700 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 29 21:59:41.209776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.209704 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 21:59:41.210356 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.210342 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b804c076-6c83-47d5-8fc1-0b259492dee4" containerName="revoke-keys" Oct 29 21:59:41.210408 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.210358 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b804c076-6c83-47d5-8fc1-0b259492dee4" containerName="revoke-keys" Oct 29 21:59:41.210471 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.210462 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b804c076-6c83-47d5-8fc1-0b259492dee4" containerName="revoke-keys" Oct 29 21:59:41.214210 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.214194 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.235192 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.235171 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 21:59:41.241204 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.241177 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4082f\"" Oct 29 21:59:41.241312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.241222 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4082f-dockercfg-87qvf\"" Oct 29 21:59:41.291682 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.291652 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 21:59:41.304525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.304482 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.470207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.389797 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 21:59:41.470207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.455505 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brf8l\" (UniqueName: \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.470207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.455626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.470207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.455661 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.470207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.455745 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4tt\" (UniqueName: \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.556995 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.556924 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brf8l\" (UniqueName: \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.557211 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.557183 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.557282 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.557222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.557337 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.557315 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4tt\" (UniqueName: \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.557960 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.557934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.558300 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.558276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.583349 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.583328 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4tt\" (UniqueName: \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\") pod \"payload-processing-e2e-api-a-d4082f-878696498-6269p\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.583783 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.583766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brf8l\" (UniqueName: \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\") pod \"payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.632085 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.632043 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:41.832828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.832742 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:41.957495 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:41.957459 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 21:59:42.033354 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.033329 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:42.033712 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.033666 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:42.034738 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.034723 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 21:59:42.153284 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.153212 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 21:59:42.525776 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.525732 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" event={"ID":"de3c2678-90bd-4423-9f4f-5d11b0b434ab","Type":"ContainerStarted","Data":"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a"} Oct 29 21:59:42.526232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.525788 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" event={"ID":"de3c2678-90bd-4423-9f4f-5d11b0b434ab","Type":"ContainerStarted","Data":"0239fc15e1de49ea902332c5fa521e01827a4130289545b5bbc6e63b5720707c"} Oct 29 21:59:42.526232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.525817 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:42.527566 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.527540 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" event={"ID":"d22bc9af-ec20-4f68-9907-558c8d2bd0e7","Type":"ContainerStarted","Data":"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c"} Oct 29 21:59:42.527670 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.527586 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" event={"ID":"d22bc9af-ec20-4f68-9907-558c8d2bd0e7","Type":"ContainerStarted","Data":"8aced9ec4e52a01c71aa02e3b474b3e819081d9840dae3eefd85b310a7193b5e"} Oct 29 21:59:42.527670 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.527621 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:42.573967 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.573928 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" podStartSLOduration=1.573915494 podStartE2EDuration="1.573915494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:42.569565027 +0000 UTC m=+1044.311495701" watchObservedRunningTime="2026-10-29 21:59:42.573915494 +0000 UTC m=+1044.315846185" Oct 29 21:59:42.614363 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.614327 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" podStartSLOduration=1.6143154229999999 podStartE2EDuration="1.614315423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:42.613302101 +0000 UTC m=+1044.355232884" watchObservedRunningTime="2026-10-29 21:59:42.614315423 +0000 UTC m=+1044.356246094" Oct 29 21:59:42.752578 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.752552 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 21:59:42.757590 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.757562 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.780056 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.779774 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d7f9e0-openshift-default-dockercfg-d2km4\"" Oct 29 21:59:42.782766 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.782733 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.782934 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.782842 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.782934 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.782892 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9zv\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783042 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.782946 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783042 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.782989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783042 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.783016 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.783045 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.783105 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.783225 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.783177 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.789126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.788783 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 21:59:42.919870 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.919836 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920357 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920342 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920491 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920478 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920654 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920850 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.920997 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.920980 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.921219 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.921203 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.921386 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.921359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9zv\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.923622 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.923248 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.939106 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.934192 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.939465 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.939409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.939958 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.939940 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.976541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.940464 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.976541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.952987 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9zv\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:42.984888 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:42.984849 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:43.012821 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.012789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:43.037752 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.037627 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\") pod \"e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:43.084590 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.084564 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:43.272035 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.272004 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn"] Oct 29 21:59:43.278030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.278003 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.294202 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.291906 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn"] Oct 29 21:59:43.296050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.296002 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:43.296050 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.296006 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:43.296322 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.296167 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:43.296418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.296211 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:43.430971 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:43.430939 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd17290b_d871_4cd2_9843_198f3d1dfe68.slice/crio-cda6553985856dcfb59c8d599942f93954548a8d2b776c79ebf7dcab3b51fa3b WatchSource:0}: Error finding container cda6553985856dcfb59c8d599942f93954548a8d2b776c79ebf7dcab3b51fa3b: Status 404 returned error can't find the container with id cda6553985856dcfb59c8d599942f93954548a8d2b776c79ebf7dcab3b51fa3b Oct 29 21:59:43.431630 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.431581 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 21:59:43.433545 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.433509 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:43.433660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.433613 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:43.433724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.433664 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:43.442176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.442154 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.442277 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.442184 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpkb\" (UniqueName: \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.442384 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.442358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.539473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.539434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerStarted","Data":"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09"} Oct 29 21:59:43.539937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.539494 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerStarted","Data":"cda6553985856dcfb59c8d599942f93954548a8d2b776c79ebf7dcab3b51fa3b"} Oct 29 21:59:43.554828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.554286 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.554828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.554561 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.554828 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.554603 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpkb\" (UniqueName: \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.558814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.556128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.559509 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.559481 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.580330 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.580309 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpkb\" (UniqueName: \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\") pod \"maas-api-revoke-sub-d64cfc2a-svhpn\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.592485 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.592446 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podStartSLOduration=1.592434274 podStartE2EDuration="1.592434274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:43.591315488 +0000 UTC m=+1045.333246165" watchObservedRunningTime="2026-10-29 21:59:43.592434274 +0000 UTC m=+1045.334364946" Oct 29 21:59:43.603964 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.603945 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:43.919758 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:43.919729 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn"] Oct 29 21:59:43.920124 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:43.920102 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1768a709_706b_47a1_ab45_c1d5713cb4f4.slice/crio-82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2 WatchSource:0}: Error finding container 82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2: Status 404 returned error can't find the container with id 82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2 Oct 29 21:59:44.084726 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.084706 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:44.089905 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.089884 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:44.550303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.550217 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" event={"ID":"1768a709-706b-47a1-ab45-c1d5713cb4f4","Type":"ContainerStarted","Data":"96534d3267bc2488a70d5a64ead92eece23ddc8abec541e7614da03e88cf3441"} Oct 29 21:59:44.550303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.550272 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" event={"ID":"1768a709-706b-47a1-ab45-c1d5713cb4f4","Type":"ContainerStarted","Data":"82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2"} Oct 29 21:59:44.550815 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.550609 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:44.551445 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.551426 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 21:59:44.599248 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:44.599207 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" podStartSLOduration=1.599193493 podStartE2EDuration="1.599193493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:44.597625079 +0000 UTC m=+1046.339555751" watchObservedRunningTime="2026-10-29 21:59:44.599193493 +0000 UTC m=+1046.341124165" Oct 29 21:59:45.561415 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:45.561382 4543 generic.go:356] "Generic (PLEG): container finished" podID="1768a709-706b-47a1-ab45-c1d5713cb4f4" containerID="96534d3267bc2488a70d5a64ead92eece23ddc8abec541e7614da03e88cf3441" exitCode=0 Oct 29 21:59:45.561840 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:45.561477 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" event={"ID":"1768a709-706b-47a1-ab45-c1d5713cb4f4","Type":"ContainerDied","Data":"96534d3267bc2488a70d5a64ead92eece23ddc8abec541e7614da03e88cf3441"} Oct 29 21:59:46.732473 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.732450 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:46.809523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.809492 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\" (UniqueName: \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\") pod \"1768a709-706b-47a1-ab45-c1d5713cb4f4\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " Oct 29 21:59:46.809694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.809639 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\") pod \"1768a709-706b-47a1-ab45-c1d5713cb4f4\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " Oct 29 21:59:46.809767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.809723 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\") pod \"1768a709-706b-47a1-ab45-c1d5713cb4f4\" (UID: \"1768a709-706b-47a1-ab45-c1d5713cb4f4\") " Oct 29 21:59:46.810137 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.810106 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca" pod "1768a709-706b-47a1-ab45-c1d5713cb4f4" (UID: "1768a709-706b-47a1-ab45-c1d5713cb4f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:46.810265 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.810216 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:46.811819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.811792 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb" pod "1768a709-706b-47a1-ab45-c1d5713cb4f4" (UID: "1768a709-706b-47a1-ab45-c1d5713cb4f4"). InnerVolumeSpecName "kube-api-access-9lpkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:46.812298 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.812275 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca" pod "1768a709-706b-47a1-ab45-c1d5713cb4f4" (UID: "1768a709-706b-47a1-ab45-c1d5713cb4f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:46.912218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.912187 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1768a709-706b-47a1-ab45-c1d5713cb4f4-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:46.912218 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:46.912212 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpkb\" (UniqueName: \"kubernetes.io/projected/1768a709-706b-47a1-ab45-c1d5713cb4f4-kube-api-access-9lpkb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:47.579722 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.579692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" event={"ID":"1768a709-706b-47a1-ab45-c1d5713cb4f4","Type":"ContainerDied","Data":"82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2"} Oct 29 21:59:47.579722 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.579715 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn" Oct 29 21:59:47.579924 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.579739 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cf2ba7c4c75eeed1196d34cc87f683e518f6184d1da251180d98ea85272dc2" Oct 29 21:59:47.919301 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.919228 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 21:59:47.919951 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.919933 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1768a709-706b-47a1-ab45-c1d5713cb4f4" containerName="revoke-keys" Oct 29 21:59:47.920030 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.919956 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1768a709-706b-47a1-ab45-c1d5713cb4f4" containerName="revoke-keys" Oct 29 21:59:47.920222 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.920208 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1768a709-706b-47a1-ab45-c1d5713cb4f4" containerName="revoke-keys" Oct 29 21:59:47.924232 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.924216 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:47.941973 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.941949 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d7f9e0\"" Oct 29 21:59:47.943541 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.943519 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 21:59:47.950338 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:47.950316 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d7f9e0-dockercfg-b45lw\"" Oct 29 21:59:48.018168 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.018137 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 21:59:48.059096 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.051476 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.067095 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.064133 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26l2\" (UniqueName: \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.067095 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.064398 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.137138 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.136715 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 21:59:48.172549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.172437 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26l2\" (UniqueName: \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.172549 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.172511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.172773 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.172754 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.172814 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.172792 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgr9l\" (UniqueName: \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.174610 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.174573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.199335 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.199313 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26l2\" (UniqueName: \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\") pod \"payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.242819 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.242543 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:48.275146 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.275109 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgr9l\" (UniqueName: \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.275316 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.275258 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.276041 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.276023 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.302732 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.302702 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgr9l\" (UniqueName: \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\") pod \"payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.450504 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.450393 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:48.577203 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.577172 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 21:59:48.579683 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:48.579653 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda541195a_7e55_4e2c_a0c4_daf3a92e11f3.slice/crio-429ad95b9174c1a57da76ddfb66033652a857302007eb5773df8bc67e1cb5cf8 WatchSource:0}: Error finding container 429ad95b9174c1a57da76ddfb66033652a857302007eb5773df8bc67e1cb5cf8: Status 404 returned error can't find the container with id 429ad95b9174c1a57da76ddfb66033652a857302007eb5773df8bc67e1cb5cf8 Oct 29 21:59:48.760378 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:48.760353 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156f8fb6_21dd_4c6b_b325_e1728d665493.slice/crio-63e0a32aadf6f26e7a59897fc839e9ce800f4054ce1291fff828de3e482f9fc9 WatchSource:0}: Error finding container 63e0a32aadf6f26e7a59897fc839e9ce800f4054ce1291fff828de3e482f9fc9: Status 404 returned error can't find the container with id 63e0a32aadf6f26e7a59897fc839e9ce800f4054ce1291fff828de3e482f9fc9 Oct 29 21:59:48.761312 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:48.761293 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 21:59:49.598661 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.598623 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" event={"ID":"a541195a-7e55-4e2c-a0c4-daf3a92e11f3","Type":"ContainerStarted","Data":"1272ea4b7ef129339c805d08256c2940b747b91c30c0f8400b85b1cb4f6e9448"} Oct 29 21:59:49.599100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.598707 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" event={"ID":"a541195a-7e55-4e2c-a0c4-daf3a92e11f3","Type":"ContainerStarted","Data":"429ad95b9174c1a57da76ddfb66033652a857302007eb5773df8bc67e1cb5cf8"} Oct 29 21:59:49.599100 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.598738 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 21:59:49.600342 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.600314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" event={"ID":"156f8fb6-21dd-4c6b-b325-e1728d665493","Type":"ContainerStarted","Data":"133e304deb1a83de1ae18222b7ff8a61b8f1c0303c95f8e98d7afc4384f737b1"} Oct 29 21:59:49.600397 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.600368 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" event={"ID":"156f8fb6-21dd-4c6b-b325-e1728d665493","Type":"ContainerStarted","Data":"63e0a32aadf6f26e7a59897fc839e9ce800f4054ce1291fff828de3e482f9fc9"} Oct 29 21:59:49.600450 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.600434 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 21:59:49.646709 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.646661 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" podStartSLOduration=2.646649291 podStartE2EDuration="2.646649291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:49.641802448 +0000 UTC m=+1051.383733123" watchObservedRunningTime="2026-10-29 21:59:49.646649291 +0000 UTC m=+1051.388579963" Oct 29 21:59:49.687112 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.687062 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" podStartSLOduration=2.687050709 podStartE2EDuration="2.687050709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:49.685123811 +0000 UTC m=+1051.427054483" watchObservedRunningTime="2026-10-29 21:59:49.687050709 +0000 UTC m=+1051.428981389" Oct 29 21:59:49.979240 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.979215 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 21:59:49.983913 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:49.983891 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.024488 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.024455 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 21:59:50.101003 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.100975 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101194 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101127 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101313 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101201 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101376 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101300 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzwj\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101376 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101340 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101481 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101436 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101536 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.101628 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.101613 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203038 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203008 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203059 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203126 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203183 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzwj\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203271 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203334 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203322 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203525 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203372 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203582 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203543 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203714 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203688 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.203842 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203730 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.204016 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203994 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.204099 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.203999 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.205490 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.205472 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.205873 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.205856 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.229340 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.229283 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzwj\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.229441 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.229337 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.307868 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.307844 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:50.643733 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.643710 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 21:59:50.644478 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:50.644447 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9987146b_2f54_4aba_8550_8b9b55e2e00c.slice/crio-a2847c75d91fb73d07eb94359d38754c692b9ea44ce20b27e8e1935297153c24 WatchSource:0}: Error finding container a2847c75d91fb73d07eb94359d38754c692b9ea44ce20b27e8e1935297153c24: Status 404 returned error can't find the container with id a2847c75d91fb73d07eb94359d38754c692b9ea44ce20b27e8e1935297153c24 Oct 29 21:59:50.647000 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.646965 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:50.647126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.647046 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:50.647126 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:50.647111 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:51.623991 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:51.623953 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" event={"ID":"9987146b-2f54-4aba-8550-8b9b55e2e00c","Type":"ContainerStarted","Data":"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f"} Oct 29 21:59:51.624287 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:51.624066 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" event={"ID":"9987146b-2f54-4aba-8550-8b9b55e2e00c","Type":"ContainerStarted","Data":"a2847c75d91fb73d07eb94359d38754c692b9ea44ce20b27e8e1935297153c24"} Oct 29 21:59:51.697502 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:51.697437 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podStartSLOduration=2.697421919 podStartE2EDuration="2.697421919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:51.682217986 +0000 UTC m=+1053.424148662" watchObservedRunningTime="2026-10-29 21:59:51.697421919 +0000 UTC m=+1053.439352591" Oct 29 21:59:52.308820 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:52.308786 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:52.310175 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:52.310149 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:52.310302 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:52.310205 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:53.308355 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:53.308318 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:53.308739 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:53.308371 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:53.554957 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:53.554930 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 21:59:53.555163 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:53.555003 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 21:59:54.057145 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.057064 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 21:59:54.064767 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.064739 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.082453 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.082424 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 21:59:54.179993 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.179962 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180170 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180014 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180176 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180233 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180203 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180368 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180350 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180404 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180385 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180478 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk887\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180522 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.180572 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.180560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282289 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282263 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282443 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282347 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282443 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282369 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282527 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282447 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282527 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282464 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282527 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282504 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk887\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282651 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282651 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282600 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282734 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282657 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282827 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282802 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282804 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.282937 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.282883 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.283134 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.283113 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.283464 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.283442 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.285028 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.285006 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.285179 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.285135 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.308272 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.308211 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.308272 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.308231 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:54.308424 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.308273 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:54.308748 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.308557 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk887\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.388970 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.388932 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:54.758631 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:54.758596 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e6fe73_41ef_41a4_8185_88a1f7802eaf.slice/crio-fa74e4cdf2251b6346ddbeb3114cbab3db49c18f6234cb3ebf54850a70bb2315 WatchSource:0}: Error finding container fa74e4cdf2251b6346ddbeb3114cbab3db49c18f6234cb3ebf54850a70bb2315: Status 404 returned error can't find the container with id fa74e4cdf2251b6346ddbeb3114cbab3db49c18f6234cb3ebf54850a70bb2315 Oct 29 21:59:54.759294 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.759261 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 21:59:54.761434 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.761403 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:54.761510 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.761469 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:54.761560 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:54.761510 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 21:59:55.308632 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.308601 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:55.309025 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.308661 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:55.358482 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.358449 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8"] Oct 29 21:59:55.363198 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.363181 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.379660 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.379633 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8"] Oct 29 21:59:55.380490 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.380460 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:59:55.386791 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.386770 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:59:55.386929 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.386770 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:59:55.387015 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.386774 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:59:55.500877 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.500845 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.501263 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.501236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcg5\" (UniqueName: \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.501640 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.501616 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.603480 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.603391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.603480 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.603455 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcg5\" (UniqueName: \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.603724 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.603591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.604240 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.604211 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.605943 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.605917 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.655160 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.655129 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcg5\" (UniqueName: \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\") pod \"maas-api-revoke-sub-40db6d2f-924j8\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.679587 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.679560 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:55.690516 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.690491 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" event={"ID":"41e6fe73-41ef-41a4-8185-88a1f7802eaf","Type":"ContainerStarted","Data":"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e"} Oct 29 21:59:55.690626 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.690536 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" event={"ID":"41e6fe73-41ef-41a4-8185-88a1f7802eaf","Type":"ContainerStarted","Data":"fa74e4cdf2251b6346ddbeb3114cbab3db49c18f6234cb3ebf54850a70bb2315"} Oct 29 21:59:55.760755 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:55.760686 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podStartSLOduration=1.760665999 podStartE2EDuration="1.760665999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.741630806 +0000 UTC m=+1057.483561485" watchObservedRunningTime="2026-10-29 21:59:55.760665999 +0000 UTC m=+1057.502596673" Oct 29 21:59:56.082603 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.082569 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8"] Oct 29 21:59:56.084443 ip-10-0-128-100 kubenswrapper[4543]: W1029 21:59:56.084417 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8b12e0_9103_4f6e_9d1f_70800c12a353.slice/crio-99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9 WatchSource:0}: Error finding container 99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9: Status 404 returned error can't find the container with id 99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9 Oct 29 21:59:56.309207 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.309169 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.309552 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.309229 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:56.393407 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.392660 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.393407 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.392706 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 21:59:56.401831 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.401803 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 21:59:56.699605 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.699526 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" event={"ID":"5a8b12e0-9103-4f6e-9d1f-70800c12a353","Type":"ContainerStarted","Data":"561e6f3d43d8182c830b406c596dd9f875069b07236620e044829bbdafcd57a0"} Oct 29 21:59:56.699804 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.699781 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" event={"ID":"5a8b12e0-9103-4f6e-9d1f-70800c12a353","Type":"ContainerStarted","Data":"99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9"} Oct 29 21:59:56.769134 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:56.767922 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" podStartSLOduration=1.767898934 podStartE2EDuration="1.767898934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:56.743756839 +0000 UTC m=+1058.485687513" watchObservedRunningTime="2026-10-29 21:59:56.767898934 +0000 UTC m=+1058.509829610" Oct 29 21:59:57.308410 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.308383 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.308523 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.308431 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 21:59:57.389973 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.389940 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.390418 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.390001 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 21:59:57.708862 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.708774 4543 generic.go:356] "Generic (PLEG): container finished" podID="5a8b12e0-9103-4f6e-9d1f-70800c12a353" containerID="561e6f3d43d8182c830b406c596dd9f875069b07236620e044829bbdafcd57a0" exitCode=0 Oct 29 21:59:57.709013 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:57.708851 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" event={"ID":"5a8b12e0-9103-4f6e-9d1f-70800c12a353","Type":"ContainerDied","Data":"561e6f3d43d8182c830b406c596dd9f875069b07236620e044829bbdafcd57a0"} Oct 29 21:59:57.991408 ip-10-0-128-100 kubenswrapper[4543]: E1029 21:59:57.991325 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 21:59:58.390043 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.389951 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.390043 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.390001 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 21:59:58.527994 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.527964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:58.528911 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.528890 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:58.539319 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.539296 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 21:59:58.971317 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:58.971254 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:59.094694 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.094656 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\") pod \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " Oct 29 21:59:59.094857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.094754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\") pod \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " Oct 29 21:59:59.094857 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.094805 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\" (UniqueName: \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\") pod \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\" (UID: \"5a8b12e0-9103-4f6e-9d1f-70800c12a353\") " Oct 29 21:59:59.095237 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.095201 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca" pod "5a8b12e0-9103-4f6e-9d1f-70800c12a353" (UID: "5a8b12e0-9103-4f6e-9d1f-70800c12a353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:59:59.095805 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.095768 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:59.097176 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.097135 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca" pod "5a8b12e0-9103-4f6e-9d1f-70800c12a353" (UID: "5a8b12e0-9103-4f6e-9d1f-70800c12a353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:59:59.097927 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.097898 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5" pod "5a8b12e0-9103-4f6e-9d1f-70800c12a353" (UID: "5a8b12e0-9103-4f6e-9d1f-70800c12a353"). InnerVolumeSpecName "kube-api-access-rzcg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:59:59.197069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.197029 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8b12e0-9103-4f6e-9d1f-70800c12a353-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:59.197069 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.197057 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzcg5\" (UniqueName: \"kubernetes.io/projected/5a8b12e0-9103-4f6e-9d1f-70800c12a353-kube-api-access-rzcg5\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 21:59:59.389303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.389216 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 21:59:59.389303 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.389268 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 21:59:59.727340 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.727314 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" Oct 29 21:59:59.727716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.727346 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8" event={"ID":"5a8b12e0-9103-4f6e-9d1f-70800c12a353","Type":"ContainerDied","Data":"99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9"} Oct 29 21:59:59.727716 ip-10-0-128-100 kubenswrapper[4543]: I1029 21:59:59.727395 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f7579ef23885f4d9a2ec8cfe4465144b421551aa4dc2c7aa2f92e7356db5c9" Oct 29 22:00:00.165148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.164887 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:00.166097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.165834 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5a8b12e0-9103-4f6e-9d1f-70800c12a353" containerName="revoke-keys" Oct 29 22:00:00.166097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.165855 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8b12e0-9103-4f6e-9d1f-70800c12a353" containerName="revoke-keys" Oct 29 22:00:00.166097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.166031 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8b12e0-9103-4f6e-9d1f-70800c12a353" containerName="revoke-keys" Oct 29 22:00:00.170345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.170326 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.170848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.170831 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:00.175826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.175803 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.176008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.175937 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:00.197138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.194530 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:00.197138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.194657 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.197138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.196409 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:00:00.197138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.196520 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:00:00.197138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.196592 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:00:00.205889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.205867 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:00.206025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.206002 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.212509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.211095 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:00.212509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.211146 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:00.212509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.211170 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:00.212509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.211250 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.220386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.219054 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:00.224318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.224092 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:00.270768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.269872 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:00.270768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.270000 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.292265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.292243 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:00.292512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.292494 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.301737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.301710 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:00.315652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.315213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.315652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.315265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kvp\" (UniqueName: \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.315652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.315360 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn52f\" (UniqueName: \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.315652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.315393 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rcq\" (UniqueName: \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.315652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.315511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.316493 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tg9f\" (UniqueName: \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.316547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.316621 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.316828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpm85\" (UniqueName: \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.316880 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.318518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.318450 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:00.390244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.390192 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:00:00.392134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.391493 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 22:00:00.418112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418000 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tg9f\" (UniqueName: \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.418112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.418319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418150 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.418319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418239 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnqh\" (UniqueName: \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.418319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418311 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.418482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpm85\" (UniqueName: \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.418482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418425 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25xw\" (UniqueName: \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.418482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418465 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.418639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.418639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418543 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kvp\" (UniqueName: \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.418639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418625 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.418793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418660 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn52f\" (UniqueName: \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.418793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418693 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rcq\" (UniqueName: \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.418907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.418907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.418859 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.419024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.419001 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.419568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.419544 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.420325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.419868 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.420325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.420178 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.475023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.474958 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn52f\" (UniqueName: \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.476276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.475759 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tg9f\" (UniqueName: \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\") pod \"maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.476531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.476489 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kvp\" (UniqueName: \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.478994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.478973 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpm85\" (UniqueName: \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\") pod \"maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.479949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.479925 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rcq\" (UniqueName: \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.492520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.492498 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:00.506103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.506063 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:00.520657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.520627 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnqh\" (UniqueName: \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.520784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.520724 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.520851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.520809 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25xw\" (UniqueName: \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.520937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.520920 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.521571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.521548 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.522218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.522196 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.563328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.563285 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnqh\" (UniqueName: \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\") pod \"maas-api-key-cleanup-29888520-pttzl\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.566113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.566021 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:00.568495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.568462 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25xw\" (UniqueName: \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:00.582041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.582007 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:00.593700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.593520 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:00.602674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.602508 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:00.612577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.612519 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 22:00:00.616068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.616046 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 22:00:00.712502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:00.712462 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:01.060457 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:01.060424 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f49517_6da7_4143_88b6_e5330fc0f31d.slice/crio-721c988ca6a0e381a2b86e69b1a4f1590d9054296161693bf7f6aa065b1c62fb WatchSource:0}: Error finding container 721c988ca6a0e381a2b86e69b1a4f1590d9054296161693bf7f6aa065b1c62fb: Status 404 returned error can't find the container with id 721c988ca6a0e381a2b86e69b1a4f1590d9054296161693bf7f6aa065b1c62fb Oct 29 22:00:01.061162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.061116 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:01.072670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.072640 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:01.073406 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:01.073275 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice/crio-d01543c4481a461e206c79f33e30a6b2d75d92f18d498c7517dd700461e4e698 WatchSource:0}: Error finding container d01543c4481a461e206c79f33e30a6b2d75d92f18d498c7517dd700461e4e698: Status 404 returned error can't find the container with id d01543c4481a461e206c79f33e30a6b2d75d92f18d498c7517dd700461e4e698 Oct 29 22:00:01.158575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.158536 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:01.165151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.165128 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:01.194061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.194040 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:01.221114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.221091 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:01.246814 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:01.246690 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd6e0e2_699f_405a_b95e_8752f25ad485.slice/crio-c5bf0f67c8cc3b70b3aa03e5b08f6bf689e2227c28c83feccccd7381c821f032 WatchSource:0}: Error finding container c5bf0f67c8cc3b70b3aa03e5b08f6bf689e2227c28c83feccccd7381c821f032: Status 404 returned error can't find the container with id c5bf0f67c8cc3b70b3aa03e5b08f6bf689e2227c28c83feccccd7381c821f032 Oct 29 22:00:01.247029 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:01.247008 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7ba4f8_16b9_4370_ba78_71d8c2a09bfd.slice/crio-6fe0459ac04d90a5270032e18a7e8d90f6fc72d1ae20dcdb637fd76cabb6dc85 WatchSource:0}: Error finding container 6fe0459ac04d90a5270032e18a7e8d90f6fc72d1ae20dcdb637fd76cabb6dc85: Status 404 returned error can't find the container with id 6fe0459ac04d90a5270032e18a7e8d90f6fc72d1ae20dcdb637fd76cabb6dc85 Oct 29 22:00:01.254709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.254690 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:01.330098 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:01.330048 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e4fcfc_d492_4b4b_b035_b8199dba76ad.slice/crio-474d5ccec5896fc1c955f70e11c25ec4afb5bdd7f0a0defd5ddcefc8652b662b WatchSource:0}: Error finding container 474d5ccec5896fc1c955f70e11c25ec4afb5bdd7f0a0defd5ddcefc8652b662b: Status 404 returned error can't find the container with id 474d5ccec5896fc1c955f70e11c25ec4afb5bdd7f0a0defd5ddcefc8652b662b Oct 29 22:00:01.389553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.389527 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:00:01.389651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.389571 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 29 22:00:01.809340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.809306 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerStarted","Data":"3c7b07f557bef158089b7a66fe8a74f8db233a62588cbbac7c670f6af4d33195"} Oct 29 22:00:01.810828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.810801 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerStarted","Data":"c5bf0f67c8cc3b70b3aa03e5b08f6bf689e2227c28c83feccccd7381c821f032"} Oct 29 22:00:01.812436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.812412 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerStarted","Data":"721c988ca6a0e381a2b86e69b1a4f1590d9054296161693bf7f6aa065b1c62fb"} Oct 29 22:00:01.813888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.813865 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerStarted","Data":"474d5ccec5896fc1c955f70e11c25ec4afb5bdd7f0a0defd5ddcefc8652b662b"} Oct 29 22:00:01.815310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.815284 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerStarted","Data":"4a0fc1e81666932dd1d01530d50ef223a2b26628a0683a4666ec2fc3cb4f049a"} Oct 29 22:00:01.816702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.816680 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerStarted","Data":"6fe0459ac04d90a5270032e18a7e8d90f6fc72d1ae20dcdb637fd76cabb6dc85"} Oct 29 22:00:01.818574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.818550 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerStarted","Data":"c774c919d876f264a8ec29e2c682eb2fc38628034510c9b40bccdfaa4fd5da7a"} Oct 29 22:00:01.818666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.818594 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerStarted","Data":"d01543c4481a461e206c79f33e30a6b2d75d92f18d498c7517dd700461e4e698"} Oct 29 22:00:01.863985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:01.863924 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" podStartSLOduration=1.8639114129999999 podStartE2EDuration="1.863911413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:01.861069678 +0000 UTC m=+1063.603000351" watchObservedRunningTime="2026-10-29 22:00:01.863911413 +0000 UTC m=+1063.605842116" Oct 29 22:00:02.805449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.805419 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 22:00:02.807130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.805929 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 22:00:02.807130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.806937 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 22:00:02.831598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.831538 4543 generic.go:356] "Generic (PLEG): container finished" podID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerID="71a1f196679f73e2a4f74d3468f92c127bc4da7a766b80d0ee81bc5401fc6f70" exitCode=7 Oct 29 22:00:02.831972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.831951 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerDied","Data":"71a1f196679f73e2a4f74d3468f92c127bc4da7a766b80d0ee81bc5401fc6f70"} Oct 29 22:00:02.832207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.832191 4543 scope.go:122] "RemoveContainer" containerID="71a1f196679f73e2a4f74d3468f92c127bc4da7a766b80d0ee81bc5401fc6f70" Oct 29 22:00:02.833784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.833758 4543 generic.go:356] "Generic (PLEG): container finished" podID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerID="6e710cee9431dea2fb2652ed4846f8e8ca03aa4d7743262e0067cb2f346e1451" exitCode=60 Oct 29 22:00:02.833877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.833860 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerDied","Data":"6e710cee9431dea2fb2652ed4846f8e8ca03aa4d7743262e0067cb2f346e1451"} Oct 29 22:00:02.834218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.834200 4543 scope.go:122] "RemoveContainer" containerID="6e710cee9431dea2fb2652ed4846f8e8ca03aa4d7743262e0067cb2f346e1451" Oct 29 22:00:02.836051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.836008 4543 generic.go:356] "Generic (PLEG): container finished" podID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerID="8f000ac2b4a6b697686e13af4596ef51aedae4cf738f6323e89830e0d7d6d1b5" exitCode=60 Oct 29 22:00:02.836142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.836094 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerDied","Data":"8f000ac2b4a6b697686e13af4596ef51aedae4cf738f6323e89830e0d7d6d1b5"} Oct 29 22:00:02.836410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.836380 4543 scope.go:122] "RemoveContainer" containerID="8f000ac2b4a6b697686e13af4596ef51aedae4cf738f6323e89830e0d7d6d1b5" Oct 29 22:00:02.837835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.837797 4543 generic.go:356] "Generic (PLEG): container finished" podID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerID="720fe66fc8b5b2dc227f98886876ec18eba5579926f27ed12912d444a197c207" exitCode=60 Oct 29 22:00:02.837883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.837870 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerDied","Data":"720fe66fc8b5b2dc227f98886876ec18eba5579926f27ed12912d444a197c207"} Oct 29 22:00:02.838045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.838029 4543 scope.go:122] "RemoveContainer" containerID="720fe66fc8b5b2dc227f98886876ec18eba5579926f27ed12912d444a197c207" Oct 29 22:00:02.839764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.839745 4543 generic.go:356] "Generic (PLEG): container finished" podID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerID="75136695c82dffb6a0c54cf810272267cc304411c49edb1ca5ca28e00c00d43d" exitCode=60 Oct 29 22:00:02.839842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.839798 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerDied","Data":"75136695c82dffb6a0c54cf810272267cc304411c49edb1ca5ca28e00c00d43d"} Oct 29 22:00:02.840054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.840038 4543 scope.go:122] "RemoveContainer" containerID="75136695c82dffb6a0c54cf810272267cc304411c49edb1ca5ca28e00c00d43d" Oct 29 22:00:02.841899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.841869 4543 generic.go:356] "Generic (PLEG): container finished" podID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerID="c774c919d876f264a8ec29e2c682eb2fc38628034510c9b40bccdfaa4fd5da7a" exitCode=60 Oct 29 22:00:02.842011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.841947 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerDied","Data":"c774c919d876f264a8ec29e2c682eb2fc38628034510c9b40bccdfaa4fd5da7a"} Oct 29 22:00:02.842183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.842166 4543 scope.go:122] "RemoveContainer" containerID="c774c919d876f264a8ec29e2c682eb2fc38628034510c9b40bccdfaa4fd5da7a" Oct 29 22:00:02.843676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.843656 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerStarted","Data":"7454d0fa4df5a59ec579efd95bd344870dddbb3a3eeb8540e628f75196fcc5e7"} Oct 29 22:00:02.918347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:02.918306 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" podStartSLOduration=2.9182919480000002 podStartE2EDuration="2.918291948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.915580706 +0000 UTC m=+1064.657511380" watchObservedRunningTime="2026-10-29 22:00:02.918291948 +0000 UTC m=+1064.660222622" Oct 29 22:00:03.955102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:03.950273 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerStarted","Data":"bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959"} Oct 29 22:00:04.043810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.041544 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" podStartSLOduration=4.041524664 podStartE2EDuration="4.041524664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:04.038423419 +0000 UTC m=+1065.780354096" watchObservedRunningTime="2026-10-29 22:00:04.041524664 +0000 UTC m=+1065.783455338" Oct 29 22:00:04.123940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.118413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerStarted","Data":"986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a"} Oct 29 22:00:04.128508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.128101 4543 generic.go:356] "Generic (PLEG): container finished" podID="487ed603-911a-430b-9252-ba9135ac8e7b" containerID="7454d0fa4df5a59ec579efd95bd344870dddbb3a3eeb8540e628f75196fcc5e7" exitCode=60 Oct 29 22:00:04.128508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.128201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerDied","Data":"7454d0fa4df5a59ec579efd95bd344870dddbb3a3eeb8540e628f75196fcc5e7"} Oct 29 22:00:04.136736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.136318 4543 scope.go:122] "RemoveContainer" containerID="7454d0fa4df5a59ec579efd95bd344870dddbb3a3eeb8540e628f75196fcc5e7" Oct 29 22:00:04.294101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.269132 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p"] Oct 29 22:00:04.314332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.313286 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6"] Oct 29 22:00:04.324701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.324439 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1960c30d-hxv2p"] Oct 29 22:00:04.364100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.364017 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd4b73c5-8tfw6"] Oct 29 22:00:04.741574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.741539 4543 scope.go:122] "RemoveContainer" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" Oct 29 22:00:04.771156 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:04.748797 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7_odh-ai-gateway-infra(6dd6e0e2-699f-405a-b95e-8752f25ad485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" Oct 29 22:00:04.868102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.860163 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa" path="/var/lib/kubelet/pods/601d7b0c-7f0a-4c3b-a524-b698fb6ba2aa/volumes" Oct 29 22:00:04.868102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.861831 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a18f14-5725-400c-b663-f556bee024f8" path="/var/lib/kubelet/pods/e6a18f14-5725-400c-b663-f556bee024f8/volumes" Oct 29 22:00:04.936100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.928893 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerStarted","Data":"1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c"} Oct 29 22:00:04.948109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.944669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerStarted","Data":"150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64"} Oct 29 22:00:04.953096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:04.949249 4543 scope.go:122] "RemoveContainer" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" Oct 29 22:00:04.997574 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:04.997533 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5_odh-ai-gateway-infra(45f49517-6da7-4143-88b6-e5330fc0f31d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" Oct 29 22:00:05.042672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.042596 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerStarted","Data":"5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b"} Oct 29 22:00:05.044613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.044577 4543 scope.go:122] "RemoveContainer" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" Oct 29 22:00:05.045220 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:05.045193 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264_odh-ai-gateway-infra(c7e4fcfc-d492-4b4b-b035-b8199dba76ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" Oct 29 22:00:05.126104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.119252 4543 scope.go:122] "RemoveContainer" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" Oct 29 22:00:05.126104 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:05.119749 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9_odh-ai-gateway-infra(c256fea5-9ad9-44cd-8b47-c83dade25686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" Oct 29 22:00:05.126104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.119804 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerStarted","Data":"829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f"} Oct 29 22:00:05.310890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.310795 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:00:05.311540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.311150 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 29 22:00:05.359101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.353866 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.480208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.480164 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:00:05.708364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.708319 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.720182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.720143 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.978225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.978177 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.005120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.983898 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:00:06.005120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:05.990314 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:00:06.197551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.197517 4543 generic.go:356] "Generic (PLEG): container finished" podID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerID="bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959" exitCode=60 Oct 29 22:00:06.198676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.198647 4543 scope.go:122] "RemoveContainer" containerID="bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959" Oct 29 22:00:06.198993 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.198964 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd_odh-ai-gateway-infra(6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" Oct 29 22:00:06.199142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.198648 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerDied","Data":"bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959"} Oct 29 22:00:06.203271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.202615 4543 generic.go:356] "Generic (PLEG): container finished" podID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerID="986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a" exitCode=60 Oct 29 22:00:06.203271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.202868 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerDied","Data":"986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a"} Oct 29 22:00:06.218780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.218759 4543 scope.go:122] "RemoveContainer" containerID="75136695c82dffb6a0c54cf810272267cc304411c49edb1ca5ca28e00c00d43d" Oct 29 22:00:06.219037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.219022 4543 scope.go:122] "RemoveContainer" containerID="986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a" Oct 29 22:00:06.219645 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.219620 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7_odh-ai-gateway-infra(20113d62-44cf-41ff-af86-a9a86e5dad94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" Oct 29 22:00:06.239225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.231164 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerStarted","Data":"d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb"} Oct 29 22:00:06.248147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.246579 4543 generic.go:356] "Generic (PLEG): container finished" podID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" exitCode=60 Oct 29 22:00:06.248147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.246655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerDied","Data":"1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c"} Oct 29 22:00:06.248147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.247107 4543 scope.go:122] "RemoveContainer" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" Oct 29 22:00:06.248147 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.247437 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7_odh-ai-gateway-infra(6dd6e0e2-699f-405a-b95e-8752f25ad485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" Oct 29 22:00:06.251409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.251387 4543 generic.go:356] "Generic (PLEG): container finished" podID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" exitCode=60 Oct 29 22:00:06.251723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.251702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerDied","Data":"150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64"} Oct 29 22:00:06.257446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.257429 4543 scope.go:122] "RemoveContainer" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" Oct 29 22:00:06.257998 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.257975 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5_odh-ai-gateway-infra(45f49517-6da7-4143-88b6-e5330fc0f31d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" Oct 29 22:00:06.261627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.260818 4543 generic.go:356] "Generic (PLEG): container finished" podID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" exitCode=60 Oct 29 22:00:06.261627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.260894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerDied","Data":"5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b"} Oct 29 22:00:06.261627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.261292 4543 scope.go:122] "RemoveContainer" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" Oct 29 22:00:06.261627 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.261572 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264_odh-ai-gateway-infra(c7e4fcfc-d492-4b4b-b035-b8199dba76ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" Oct 29 22:00:06.268802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.264332 4543 generic.go:356] "Generic (PLEG): container finished" podID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" exitCode=60 Oct 29 22:00:06.268802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.264381 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerDied","Data":"829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f"} Oct 29 22:00:06.268802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.264750 4543 scope.go:122] "RemoveContainer" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" Oct 29 22:00:06.268802 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:06.265011 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9_odh-ai-gateway-infra(c256fea5-9ad9-44cd-8b47-c83dade25686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" Oct 29 22:00:06.448112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.424664 4543 scope.go:122] "RemoveContainer" containerID="c774c919d876f264a8ec29e2c682eb2fc38628034510c9b40bccdfaa4fd5da7a" Oct 29 22:00:06.572239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.572144 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:00:06.684891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.681820 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.708816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.708788 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:00:06.718835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.713914 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8867e7d\"/\"e2e-lifecycle-model-f8867e7d-kserve-self-signed-certs\"" Oct 29 22:00:06.718987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.714639 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8867e7d\"/\"openshift-service-ca.crt\"" Oct 29 22:00:06.719100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.714729 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8867e7d\"/\"kube-root-ca.crt\"" Oct 29 22:00:06.719190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.714814 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8867e7d\"/\"default-dockercfg-7cqbx\"" Oct 29 22:00:06.719290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.717397 4543 scope.go:122] "RemoveContainer" containerID="71a1f196679f73e2a4f74d3468f92c127bc4da7a766b80d0ee81bc5401fc6f70" Oct 29 22:00:06.752054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.751937 4543 scope.go:122] "RemoveContainer" containerID="6e710cee9431dea2fb2652ed4846f8e8ca03aa4d7743262e0067cb2f346e1451" Oct 29 22:00:06.770196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.770176 4543 scope.go:122] "RemoveContainer" containerID="8f000ac2b4a6b697686e13af4596ef51aedae4cf738f6323e89830e0d7d6d1b5" Oct 29 22:00:06.797453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.797342 4543 scope.go:122] "RemoveContainer" containerID="720fe66fc8b5b2dc227f98886876ec18eba5579926f27ed12912d444a197c207" Oct 29 22:00:06.813980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.813956 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.814234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.814211 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.814443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.814401 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.814443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.814433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9pb\" (UniqueName: \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.814556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.814456 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.814613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.814604 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.915975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.915880 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.916159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916017 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.916159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916052 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9pb\" (UniqueName: \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.916159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.916342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.916411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916362 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.917292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.916938 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.917292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.917171 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.917292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.917261 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.918578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.918551 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.919193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.919174 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:06.946770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:06.946714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9pb\" (UniqueName: \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\") pod \"e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:07.056542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.056505 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:07.275921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.275884 4543 scope.go:122] "RemoveContainer" containerID="bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959" Oct 29 22:00:07.276306 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.276269 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd_odh-ai-gateway-infra(6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" Oct 29 22:00:07.277831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.277807 4543 scope.go:122] "RemoveContainer" containerID="986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a" Oct 29 22:00:07.278197 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.278174 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7_odh-ai-gateway-infra(20113d62-44cf-41ff-af86-a9a86e5dad94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" Oct 29 22:00:07.279741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.279717 4543 generic.go:356] "Generic (PLEG): container finished" podID="487ed603-911a-430b-9252-ba9135ac8e7b" containerID="d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb" exitCode=60 Oct 29 22:00:07.279872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.279797 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerDied","Data":"d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb"} Oct 29 22:00:07.279949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.279887 4543 scope.go:122] "RemoveContainer" containerID="7454d0fa4df5a59ec579efd95bd344870dddbb3a3eeb8540e628f75196fcc5e7" Oct 29 22:00:07.280069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.280049 4543 scope.go:122] "RemoveContainer" containerID="d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb" Oct 29 22:00:07.280397 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.280371 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-pttzl_odh-ai-gateway-infra(487ed603-911a-430b-9252-ba9135ac8e7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" Oct 29 22:00:07.281924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.281905 4543 scope.go:122] "RemoveContainer" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" Oct 29 22:00:07.282214 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.282195 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7_odh-ai-gateway-infra(6dd6e0e2-699f-405a-b95e-8752f25ad485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" Oct 29 22:00:07.283545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.283528 4543 scope.go:122] "RemoveContainer" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" Oct 29 22:00:07.283785 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.283761 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5_odh-ai-gateway-infra(45f49517-6da7-4143-88b6-e5330fc0f31d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" Oct 29 22:00:07.285340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.285216 4543 scope.go:122] "RemoveContainer" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" Oct 29 22:00:07.285626 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.285604 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264_odh-ai-gateway-infra(c7e4fcfc-d492-4b4b-b035-b8199dba76ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" Oct 29 22:00:07.287194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.287174 4543 scope.go:122] "RemoveContainer" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" Oct 29 22:00:07.287442 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:07.287418 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9_odh-ai-gateway-infra(c256fea5-9ad9-44cd-8b47-c83dade25686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" Oct 29 22:00:07.393217 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:07.392969 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb32330_e422_4ade_9002_8a6f190c48a5.slice/crio-cbb35abe48ed20c26ffdb40aaea624e711fcf325db2b62abdefbcf373834ecbf WatchSource:0}: Error finding container cbb35abe48ed20c26ffdb40aaea624e711fcf325db2b62abdefbcf373834ecbf: Status 404 returned error can't find the container with id cbb35abe48ed20c26ffdb40aaea624e711fcf325db2b62abdefbcf373834ecbf Oct 29 22:00:07.396496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:07.396470 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:00:08.297494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:08.297467 4543 scope.go:122] "RemoveContainer" containerID="d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb" Oct 29 22:00:08.297922 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:08.297810 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-pttzl_odh-ai-gateway-infra(487ed603-911a-430b-9252-ba9135ac8e7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" Oct 29 22:00:08.299146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:08.299122 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" event={"ID":"4bb32330-e422-4ade-9002-8a6f190c48a5","Type":"ContainerStarted","Data":"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e"} Oct 29 22:00:08.299259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:08.299158 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" event={"ID":"4bb32330-e422-4ade-9002-8a6f190c48a5","Type":"ContainerStarted","Data":"cbb35abe48ed20c26ffdb40aaea624e711fcf325db2b62abdefbcf373834ecbf"} Oct 29 22:00:08.391713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:08.391656 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" podStartSLOduration=2.391638856 podStartE2EDuration="2.391638856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:08.384279452 +0000 UTC m=+1070.126210121" watchObservedRunningTime="2026-10-29 22:00:08.391638856 +0000 UTC m=+1070.133569530" Oct 29 22:00:09.057986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:09.057951 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:15.378391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:15.378355 4543 scope.go:122] "RemoveContainer" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" Oct 29 22:00:15.378835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:15.378513 4543 scope.go:122] "RemoveContainer" containerID="986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a" Oct 29 22:00:16.378050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.378013 4543 scope.go:122] "RemoveContainer" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" Oct 29 22:00:16.378352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.378333 4543 scope.go:122] "RemoveContainer" containerID="bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959" Oct 29 22:00:16.378501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.378479 4543 scope.go:122] "RemoveContainer" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" Oct 29 22:00:16.382261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.382229 4543 generic.go:356] "Generic (PLEG): container finished" podID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerID="617f03cd5ba68aede3609c66915b5a7d2c5b240d10f5e3aecf6fd4b807c6467d" exitCode=60 Oct 29 22:00:16.384475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.384450 4543 generic.go:356] "Generic (PLEG): container finished" podID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerID="4afa70716e5e1b49382fb2de356c5adac24e62b26850354dbc3ea31f51fecad3" exitCode=60 Oct 29 22:00:16.393346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.393322 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerDied","Data":"617f03cd5ba68aede3609c66915b5a7d2c5b240d10f5e3aecf6fd4b807c6467d"} Oct 29 22:00:16.393451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.393367 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerDied","Data":"4afa70716e5e1b49382fb2de356c5adac24e62b26850354dbc3ea31f51fecad3"} Oct 29 22:00:16.393451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.393419 4543 scope.go:122] "RemoveContainer" containerID="5043e83b49306a8a16af0af20117973b870ed02bee4218d53e1c27d83566ae4b" Oct 29 22:00:16.393704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.393687 4543 scope.go:122] "RemoveContainer" containerID="4afa70716e5e1b49382fb2de356c5adac24e62b26850354dbc3ea31f51fecad3" Oct 29 22:00:16.393854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.393837 4543 scope.go:122] "RemoveContainer" containerID="617f03cd5ba68aede3609c66915b5a7d2c5b240d10f5e3aecf6fd4b807c6467d" Oct 29 22:00:16.393987 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:16.393967 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7_odh-ai-gateway-infra(20113d62-44cf-41ff-af86-a9a86e5dad94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" Oct 29 22:00:16.394189 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:16.394165 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264_odh-ai-gateway-infra(c7e4fcfc-d492-4b4b-b035-b8199dba76ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" Oct 29 22:00:16.439236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.439210 4543 scope.go:122] "RemoveContainer" containerID="986a5985317b939056db7d6de90505d653ec17e5f8aff5cb9b5e9bb356b7c14a" Oct 29 22:00:16.754942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.754912 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:00:16.760978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.760962 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.802604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.802570 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:00:16.880065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880039 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880141 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880195 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880281 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880409 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880445 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880574 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjnq\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.880637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.880620 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982385 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982483 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982507 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982560 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjnq\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982697 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982740 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982783 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.982959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.982916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.983030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.983009 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.983356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.983334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.983603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.983372 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.983603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.983401 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.984925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.984905 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:16.985034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:16.985014 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:17.010744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.010718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:17.010852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.010797 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjnq\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:17.058205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.058183 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:17.071409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.071385 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:17.075622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.075605 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:17.378578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.378552 4543 scope.go:122] "RemoveContainer" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" Oct 29 22:00:17.395722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.395692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerStarted","Data":"dfc422b8b5155d3675209f6b91efd75c02fb88c9a1cf2818cd2e1490b70c02c8"} Oct 29 22:00:17.399974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.399944 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerStarted","Data":"9824b21161f7120c0db9911d239e10caf93bda777abbadb6ca4221b67132c01c"} Oct 29 22:00:17.403880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.403859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerStarted","Data":"ff88a5fb9be538483c8ae01bf0c79cc819b1e87ccc05a770fda85c7ee627b691"} Oct 29 22:00:17.413931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.413909 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:00:17.417171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.417144 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:17.417508 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:17.417487 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5097e4_78c1_4d74_9759_d5198b887368.slice/crio-2fb66ec433658605ffca0e56798565318f1e2baaab8d2150a9840467e1d3d9bc WatchSource:0}: Error finding container 2fb66ec433658605ffca0e56798565318f1e2baaab8d2150a9840467e1d3d9bc: Status 404 returned error can't find the container with id 2fb66ec433658605ffca0e56798565318f1e2baaab8d2150a9840467e1d3d9bc Oct 29 22:00:17.420064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.420031 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:17.420180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.420124 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:17.420180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.420169 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:17.454892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.454864 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:17.485541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.485496 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" podStartSLOduration=17.485483271 podStartE2EDuration="17.485483271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:17.483495141 +0000 UTC m=+1079.225425813" watchObservedRunningTime="2026-10-29 22:00:17.485483271 +0000 UTC m=+1079.227413945" Oct 29 22:00:17.501955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.501919 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:17.597508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.596123 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" podStartSLOduration=17.596107362 podStartE2EDuration="17.596107362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:17.592312021 +0000 UTC m=+1079.334242695" watchObservedRunningTime="2026-10-29 22:00:17.596107362 +0000 UTC m=+1079.338038031" Oct 29 22:00:17.813637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.813610 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:17.819288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.819269 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:17.898937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.898897 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\") pod \"20113d62-44cf-41ff-af86-a9a86e5dad94\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " Oct 29 22:00:17.899240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.899094 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\") pod \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " Oct 29 22:00:17.899240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.899158 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\" (UniqueName: \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\") pod \"20113d62-44cf-41ff-af86-a9a86e5dad94\" (UID: \"20113d62-44cf-41ff-af86-a9a86e5dad94\") " Oct 29 22:00:17.899240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.899221 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\" (UniqueName: \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\") pod \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\" (UID: \"c7e4fcfc-d492-4b4b-b035-b8199dba76ad\") " Oct 29 22:00:17.899437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.899288 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca" pod "20113d62-44cf-41ff-af86-a9a86e5dad94" (UID: "20113d62-44cf-41ff-af86-a9a86e5dad94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:17.899557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.899533 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca" pod "c7e4fcfc-d492-4b4b-b035-b8199dba76ad" (UID: "c7e4fcfc-d492-4b4b-b035-b8199dba76ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:17.900033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.900007 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20113d62-44cf-41ff-af86-a9a86e5dad94-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:17.900150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.900034 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:17.902543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.902516 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f" pod "20113d62-44cf-41ff-af86-a9a86e5dad94" (UID: "20113d62-44cf-41ff-af86-a9a86e5dad94"). InnerVolumeSpecName "kube-api-access-dn52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:17.911502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:17.911476 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp" pod "c7e4fcfc-d492-4b4b-b035-b8199dba76ad" (UID: "c7e4fcfc-d492-4b4b-b035-b8199dba76ad"). InnerVolumeSpecName "kube-api-access-86kvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:18.002421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.002385 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn52f\" (UniqueName: \"kubernetes.io/projected/20113d62-44cf-41ff-af86-a9a86e5dad94-kube-api-access-dn52f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:18.002421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.002424 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86kvp\" (UniqueName: \"kubernetes.io/projected/c7e4fcfc-d492-4b4b-b035-b8199dba76ad-kube-api-access-86kvp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:18.421717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.421634 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerStarted","Data":"7dc0754a6d41500885f28e98dcc68a731567c802424f315a41465e2d5e26c54e"} Oct 29 22:00:18.447934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.441729 4543 generic.go:356] "Generic (PLEG): container finished" podID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerID="9824b21161f7120c0db9911d239e10caf93bda777abbadb6ca4221b67132c01c" exitCode=60 Oct 29 22:00:18.447934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.441931 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerDied","Data":"9824b21161f7120c0db9911d239e10caf93bda777abbadb6ca4221b67132c01c"} Oct 29 22:00:18.447934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.442000 4543 scope.go:122] "RemoveContainer" containerID="150f1d5a2c0fcde44de6ff2a29f5cb6a3a704614758d78a1db557d61dc8def64" Oct 29 22:00:18.447934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.442446 4543 scope.go:122] "RemoveContainer" containerID="9824b21161f7120c0db9911d239e10caf93bda777abbadb6ca4221b67132c01c" Oct 29 22:00:18.447934 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:18.442888 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5_odh-ai-gateway-infra(45f49517-6da7-4143-88b6-e5330fc0f31d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" Oct 29 22:00:18.506288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.504938 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" event={"ID":"c7e4fcfc-d492-4b4b-b035-b8199dba76ad","Type":"ContainerDied","Data":"474d5ccec5896fc1c955f70e11c25ec4afb5bdd7f0a0defd5ddcefc8652b662b"} Oct 29 22:00:18.506288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.505107 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264" Oct 29 22:00:18.506288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.505802 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:18.508811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.506961 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" podStartSLOduration=18.506944919 podStartE2EDuration="18.506944919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:18.47150108 +0000 UTC m=+1080.213431768" watchObservedRunningTime="2026-10-29 22:00:18.506944919 +0000 UTC m=+1080.248875594" Oct 29 22:00:18.508811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.508449 4543 generic.go:356] "Generic (PLEG): container finished" podID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerID="ff88a5fb9be538483c8ae01bf0c79cc819b1e87ccc05a770fda85c7ee627b691" exitCode=60 Oct 29 22:00:18.508811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.508547 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerDied","Data":"ff88a5fb9be538483c8ae01bf0c79cc819b1e87ccc05a770fda85c7ee627b691"} Oct 29 22:00:18.509765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.509003 4543 scope.go:122] "RemoveContainer" containerID="ff88a5fb9be538483c8ae01bf0c79cc819b1e87ccc05a770fda85c7ee627b691" Oct 29 22:00:18.509765 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:18.509441 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9_odh-ai-gateway-infra(c256fea5-9ad9-44cd-8b47-c83dade25686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" Oct 29 22:00:18.511732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.511705 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" event={"ID":"bc5097e4-78c1-4d74-9759-d5198b887368","Type":"ContainerStarted","Data":"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59"} Oct 29 22:00:18.511832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.511758 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" event={"ID":"bc5097e4-78c1-4d74-9759-d5198b887368","Type":"ContainerStarted","Data":"2fb66ec433658605ffca0e56798565318f1e2baaab8d2150a9840467e1d3d9bc"} Oct 29 22:00:18.514783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.514764 4543 generic.go:356] "Generic (PLEG): container finished" podID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerID="dfc422b8b5155d3675209f6b91efd75c02fb88c9a1cf2818cd2e1490b70c02c8" exitCode=60 Oct 29 22:00:18.514966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.514950 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerDied","Data":"dfc422b8b5155d3675209f6b91efd75c02fb88c9a1cf2818cd2e1490b70c02c8"} Oct 29 22:00:18.523800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.523600 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" Oct 29 22:00:18.523800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.523628 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7" event={"ID":"20113d62-44cf-41ff-af86-a9a86e5dad94","Type":"ContainerDied","Data":"d01543c4481a461e206c79f33e30a6b2d75d92f18d498c7517dd700461e4e698"} Oct 29 22:00:18.526199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.525547 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:18.568871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.563572 4543 scope.go:122] "RemoveContainer" containerID="617f03cd5ba68aede3609c66915b5a7d2c5b240d10f5e3aecf6fd4b807c6467d" Oct 29 22:00:18.577294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.577263 4543 scope.go:122] "RemoveContainer" containerID="9e66719dad4f97058e7f3440c9b3a55aa65777c39ae77195d312ff8c2383af7f" Oct 29 22:00:18.595537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.595518 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:18.619896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.619883 4543 scope.go:122] "RemoveContainer" containerID="829a1d9dc970feef6cd973dbb864fd00935c0d0ef2186b17a9869eb7e1f43b2f" Oct 29 22:00:18.624325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.623961 4543 scope.go:122] "RemoveContainer" containerID="c0fd0d4aa3e8771b83070fe5d859704808aafa96c1de2851eee78c976ae61725" Oct 29 22:00:18.654971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.654583 4543 scope.go:122] "RemoveContainer" containerID="3335516cd11101deaad0efbe8c32d99e61f50f04472de8264bc8088b98ff26b4" Oct 29 22:00:18.673207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.673128 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:18.677501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.677481 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888520-t6264"] Oct 29 22:00:18.683329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.683314 4543 scope.go:122] "RemoveContainer" containerID="bd8f794c147bcfa0a2ce8188a7c1eb977b22e1046a0c2be63de6609dbbe51959" Oct 29 22:00:18.719317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.719283 4543 scope.go:122] "RemoveContainer" containerID="4afa70716e5e1b49382fb2de356c5adac24e62b26850354dbc3ea31f51fecad3" Oct 29 22:00:18.766882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.766820 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podStartSLOduration=2.766797899 podStartE2EDuration="2.766797899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:18.719924683 +0000 UTC m=+1080.461855358" watchObservedRunningTime="2026-10-29 22:00:18.766797899 +0000 UTC m=+1080.508728567" Oct 29 22:00:18.791266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.791244 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:18.797385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.797362 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:18.810872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.810847 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888520-mqrw7"] Oct 29 22:00:18.922956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.922921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\") pod \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " Oct 29 22:00:18.923146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.923114 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\" (UniqueName: \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\") pod \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\" (UID: \"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd\") " Oct 29 22:00:18.923442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.923392 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca" pod "6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" (UID: "6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:18.923814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.923795 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:18.925469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:18.925445 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw" pod "6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" (UID: "6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd"). InnerVolumeSpecName "kube-api-access-x25xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:19.025601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.025574 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x25xw\" (UniqueName: \"kubernetes.io/projected/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd-kube-api-access-x25xw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:19.059439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.059400 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g"] Oct 29 22:00:19.064297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.064275 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7"] Oct 29 22:00:19.068527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.068508 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36b68b7-pt87g"] Oct 29 22:00:19.072503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.072482 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee90ee1c-94kr7"] Oct 29 22:00:19.076661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.076640 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:19.078271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.078248 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:19.078355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.078289 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:19.485971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.485942 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:19.558543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.558511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" Oct 29 22:00:19.558543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.558514 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd" event={"ID":"6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd","Type":"ContainerDied","Data":"6fe0459ac04d90a5270032e18a7e8d90f6fc72d1ae20dcdb637fd76cabb6dc85"} Oct 29 22:00:19.558798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.558578 4543 scope.go:122] "RemoveContainer" containerID="dfc422b8b5155d3675209f6b91efd75c02fb88c9a1cf2818cd2e1490b70c02c8" Oct 29 22:00:19.562411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.562384 4543 generic.go:356] "Generic (PLEG): container finished" podID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerID="7dc0754a6d41500885f28e98dcc68a731567c802424f315a41465e2d5e26c54e" exitCode=60 Oct 29 22:00:19.562545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.562472 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerDied","Data":"7dc0754a6d41500885f28e98dcc68a731567c802424f315a41465e2d5e26c54e"} Oct 29 22:00:19.574728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.574702 4543 scope.go:122] "RemoveContainer" containerID="1cbab9e30ef3a37d91cc3eb0d1c153d2cbd51e3f89b1ccfe223c688b7423a15c" Oct 29 22:00:19.754194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.750553 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:19.772128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.755732 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888520-96tsd"] Oct 29 22:00:19.838618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.834365 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:19.876812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.876784 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:19.879658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.879624 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\" (UniqueName: \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\") pod \"45f49517-6da7-4143-88b6-e5330fc0f31d\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " Oct 29 22:00:19.879778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.879763 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\") pod \"6dd6e0e2-699f-405a-b95e-8752f25ad485\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " Oct 29 22:00:19.879924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.879902 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\") pod \"45f49517-6da7-4143-88b6-e5330fc0f31d\" (UID: \"45f49517-6da7-4143-88b6-e5330fc0f31d\") " Oct 29 22:00:19.880068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.880050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\" (UniqueName: \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\") pod \"6dd6e0e2-699f-405a-b95e-8752f25ad485\" (UID: \"6dd6e0e2-699f-405a-b95e-8752f25ad485\") " Oct 29 22:00:19.881801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.881772 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca" pod "45f49517-6da7-4143-88b6-e5330fc0f31d" (UID: "45f49517-6da7-4143-88b6-e5330fc0f31d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:19.881965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.881942 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca" pod "6dd6e0e2-699f-405a-b95e-8752f25ad485" (UID: "6dd6e0e2-699f-405a-b95e-8752f25ad485"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:19.890162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.886626 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f" pod "45f49517-6da7-4143-88b6-e5330fc0f31d" (UID: "45f49517-6da7-4143-88b6-e5330fc0f31d"). InnerVolumeSpecName "kube-api-access-6tg9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:19.898551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.898445 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85" pod "6dd6e0e2-699f-405a-b95e-8752f25ad485" (UID: "6dd6e0e2-699f-405a-b95e-8752f25ad485"). InnerVolumeSpecName "kube-api-access-gpm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:19.974350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.974322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.981546 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\" (UniqueName: \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\") pod \"c256fea5-9ad9-44cd-8b47-c83dade25686\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.981754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\") pod \"c256fea5-9ad9-44cd-8b47-c83dade25686\" (UID: \"c256fea5-9ad9-44cd-8b47-c83dade25686\") " Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.982465 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpm85\" (UniqueName: \"kubernetes.io/projected/6dd6e0e2-699f-405a-b95e-8752f25ad485-kube-api-access-gpm85\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.982477 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca" pod "c256fea5-9ad9-44cd-8b47-c83dade25686" (UID: "c256fea5-9ad9-44cd-8b47-c83dade25686"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.982488 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tg9f\" (UniqueName: \"kubernetes.io/projected/45f49517-6da7-4143-88b6-e5330fc0f31d-kube-api-access-6tg9f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.982551 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6e0e2-699f-405a-b95e-8752f25ad485-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:19.982587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.982572 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49517-6da7-4143-88b6-e5330fc0f31d-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:19.984589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:19.984560 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq" pod "c256fea5-9ad9-44cd-8b47-c83dade25686" (UID: "c256fea5-9ad9-44cd-8b47-c83dade25686"). InnerVolumeSpecName "kube-api-access-86rcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:20.074005 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:20.073929 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:20.076516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.076484 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:20.076642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.076549 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:20.083807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.083786 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c256fea5-9ad9-44cd-8b47-c83dade25686-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.083807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.083808 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rcq\" (UniqueName: \"kubernetes.io/projected/c256fea5-9ad9-44cd-8b47-c83dade25686-kube-api-access-86rcq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.378203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.378116 4543 scope.go:122] "RemoveContainer" containerID="d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb" Oct 29 22:00:20.387509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.387480 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" path="/var/lib/kubelet/pods/20113d62-44cf-41ff-af86-a9a86e5dad94/volumes" Oct 29 22:00:20.387896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.387878 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" path="/var/lib/kubelet/pods/6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd/volumes" Oct 29 22:00:20.388335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.388318 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71faa10d-4060-46a6-8b9d-01180642ad24" path="/var/lib/kubelet/pods/71faa10d-4060-46a6-8b9d-01180642ad24/volumes" Oct 29 22:00:20.389059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.389042 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b624a598-0a3b-499c-9836-d9a0b8cb1b89" path="/var/lib/kubelet/pods/b624a598-0a3b-499c-9836-d9a0b8cb1b89/volumes" Oct 29 22:00:20.389448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.389430 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" path="/var/lib/kubelet/pods/c7e4fcfc-d492-4b4b-b035-b8199dba76ad/volumes" Oct 29 22:00:20.580089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.580049 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" Oct 29 22:00:20.580505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.580051 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7" event={"ID":"6dd6e0e2-699f-405a-b95e-8752f25ad485","Type":"ContainerDied","Data":"c5bf0f67c8cc3b70b3aa03e5b08f6bf689e2227c28c83feccccd7381c821f032"} Oct 29 22:00:20.580505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.580222 4543 scope.go:122] "RemoveContainer" containerID="7dc0754a6d41500885f28e98dcc68a731567c802424f315a41465e2d5e26c54e" Oct 29 22:00:20.581922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.581902 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" Oct 29 22:00:20.582018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.581921 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5" event={"ID":"45f49517-6da7-4143-88b6-e5330fc0f31d","Type":"ContainerDied","Data":"721c988ca6a0e381a2b86e69b1a4f1590d9054296161693bf7f6aa065b1c62fb"} Oct 29 22:00:20.584016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.583997 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" Oct 29 22:00:20.584353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.584324 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9" event={"ID":"c256fea5-9ad9-44cd-8b47-c83dade25686","Type":"ContainerDied","Data":"4a0fc1e81666932dd1d01530d50ef223a2b26628a0683a4666ec2fc3cb4f049a"} Oct 29 22:00:20.593087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.593057 4543 scope.go:122] "RemoveContainer" containerID="9824b21161f7120c0db9911d239e10caf93bda777abbadb6ca4221b67132c01c" Oct 29 22:00:20.602526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.602510 4543 scope.go:122] "RemoveContainer" containerID="ff88a5fb9be538483c8ae01bf0c79cc819b1e87ccc05a770fda85c7ee627b691" Oct 29 22:00:20.661742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.661666 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:20.666695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.666672 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d4082f-29888520-8vld7"] Oct 29 22:00:20.700989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.700965 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:20.706103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.706068 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a116a2-29888520-pq4z5"] Oct 29 22:00:20.758687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.758661 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:20.762995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:20.762975 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888520-8qww9"] Oct 29 22:00:20.854097 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:20.853876 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:20.854775 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:20.854747 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:21.077370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.076810 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:21.077370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.076889 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:21.594567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.594536 4543 generic.go:356] "Generic (PLEG): container finished" podID="487ed603-911a-430b-9252-ba9135ac8e7b" containerID="5cc2492b016c70fbc48d8997c09ff04547fe919d56d1eb2e39e94a14036ecd42" exitCode=60 Oct 29 22:00:21.594953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.594609 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerDied","Data":"5cc2492b016c70fbc48d8997c09ff04547fe919d56d1eb2e39e94a14036ecd42"} Oct 29 22:00:21.594953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.594683 4543 scope.go:122] "RemoveContainer" containerID="d84968a2577a058dcf57b68652ca4a99123a6ca1e95fda266bd4f1e12fb44deb" Oct 29 22:00:21.595126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:21.595101 4543 scope.go:122] "RemoveContainer" containerID="5cc2492b016c70fbc48d8997c09ff04547fe919d56d1eb2e39e94a14036ecd42" Oct 29 22:00:21.595516 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:21.595475 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-pttzl_odh-ai-gateway-infra(487ed603-911a-430b-9252-ba9135ac8e7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" Oct 29 22:00:22.075914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.075863 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:22.076090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.075945 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:22.388446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.388287 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" path="/var/lib/kubelet/pods/45f49517-6da7-4143-88b6-e5330fc0f31d/volumes" Oct 29 22:00:22.389245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.389214 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" path="/var/lib/kubelet/pods/6dd6e0e2-699f-405a-b95e-8752f25ad485/volumes" Oct 29 22:00:22.390268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.390243 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" path="/var/lib/kubelet/pods/c256fea5-9ad9-44cd-8b47-c83dade25686/volumes" Oct 29 22:00:22.656544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.656468 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:22.870125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.870097 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:22.926918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.926839 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\" (UniqueName: \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\") pod \"487ed603-911a-430b-9252-ba9135ac8e7b\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " Oct 29 22:00:22.927161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.927139 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\") pod \"487ed603-911a-430b-9252-ba9135ac8e7b\" (UID: \"487ed603-911a-430b-9252-ba9135ac8e7b\") " Oct 29 22:00:22.927615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.927588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca" pod "487ed603-911a-430b-9252-ba9135ac8e7b" (UID: "487ed603-911a-430b-9252-ba9135ac8e7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:22.927899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.927877 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487ed603-911a-430b-9252-ba9135ac8e7b-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:22.965201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:22.965165 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh" pod "487ed603-911a-430b-9252-ba9135ac8e7b" (UID: "487ed603-911a-430b-9252-ba9135ac8e7b"). InnerVolumeSpecName "kube-api-access-5dnqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:23.053529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.053495 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnqh\" (UniqueName: \"kubernetes.io/projected/487ed603-911a-430b-9252-ba9135ac8e7b-kube-api-access-5dnqh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:23.076896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.076872 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:23.076995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.076918 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:23.376206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.376172 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc"] Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.377534 4543 cpu_manager.go:414] "containerMap: removing container" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383053 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383115 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383124 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383135 4543 cpu_manager.go:414] "containerMap: removing container" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383141 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383154 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383162 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383171 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383176 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383189 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383196 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383223 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383228 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383236 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383240 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383246 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383250 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383257 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383262 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383269 4543 cpu_manager.go:414] "containerMap: removing container" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383273 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383284 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383289 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383296 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383300 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383308 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383313 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383331 4543 cpu_manager.go:414] "containerMap: removing container" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383335 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383488 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383496 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383505 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383513 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383520 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383525 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383532 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383538 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383544 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383550 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383556 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383564 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383572 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383577 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383583 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383588 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:23.385130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.383594 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:23.397457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.390324 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc"] Oct 29 22:00:23.397457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.391337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.424401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.424372 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:00:23.483099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.480103 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.483099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.480237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.483099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.480452 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjknk\" (UniqueName: \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.584099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.583793 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjknk\" (UniqueName: \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.584099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.583975 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.584099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.584093 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.585141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.584854 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.594098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.592770 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.647151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.627419 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjknk\" (UniqueName: \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\") pod \"maas-api-revoke-sub-f16b9892-vgcmc\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.667147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.661287 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" event={"ID":"487ed603-911a-430b-9252-ba9135ac8e7b","Type":"ContainerDied","Data":"3c7b07f557bef158089b7a66fe8a74f8db233a62588cbbac7c670f6af4d33195"} Oct 29 22:00:23.667147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.661365 4543 scope.go:122] "RemoveContainer" containerID="5cc2492b016c70fbc48d8997c09ff04547fe919d56d1eb2e39e94a14036ecd42" Oct 29 22:00:23.667147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.661511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl" Oct 29 22:00:23.714171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.714147 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:23.745994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.745939 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:23.753751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:23.753599 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-pttzl"] Oct 29 22:00:24.057697 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:24.057669 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ec384c_4df2_4802_a5a1_025d107be768.slice/crio-02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890 WatchSource:0}: Error finding container 02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890: Status 404 returned error can't find the container with id 02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890 Oct 29 22:00:24.058781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.058751 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc"] Oct 29 22:00:24.076141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.076114 4543 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 22:00:24.076241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.076172 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 22:00:24.160884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.160851 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:00:24.161985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.161963 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:24.162112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.161986 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd6e0e2-699f-405a-b95e-8752f25ad485" containerName="cleanup" Oct 29 22:00:24.162112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162045 4543 cpu_manager.go:414] "containerMap: removing container" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:24.162112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162053 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:24.162112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162088 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:24.162112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162096 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162129 4543 cpu_manager.go:414] "containerMap: removing container" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162136 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162154 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162161 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7ba4f8-16b9-4370-ba78-71d8c2a09bfd" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162177 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:24.162345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162183 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c256fea5-9ad9-44cd-8b47-c83dade25686" containerName="cleanup" Oct 29 22:00:24.162634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162392 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e4fcfc-d492-4b4b-b035-b8199dba76ad" containerName="cleanup" Oct 29 22:00:24.162634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162412 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49517-6da7-4143-88b6-e5330fc0f31d" containerName="cleanup" Oct 29 22:00:24.162634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162422 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="20113d62-44cf-41ff-af86-a9a86e5dad94" containerName="cleanup" Oct 29 22:00:24.162634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.162432 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" containerName="cleanup" Oct 29 22:00:24.170716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.170699 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.194996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.194965 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:00:24.297646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.297610 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.297779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.297689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzv6\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.297779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.297736 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.297900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.297847 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.297900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.297877 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.298126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.298067 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.298256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.298226 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.298343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.298307 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.298487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.298397 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.389171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.389136 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487ed603-911a-430b-9252-ba9135ac8e7b" path="/var/lib/kubelet/pods/487ed603-911a-430b-9252-ba9135ac8e7b/volumes" Oct 29 22:00:24.399143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399118 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399156 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399210 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399289 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzv6\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399333 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399385 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399411 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399558 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.399637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399623 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.400020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399954 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.400020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.399992 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.400020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.400037 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.400333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.400239 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.401600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.401574 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.402115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.401948 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.425532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.425506 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.425748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.425728 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzv6\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.496380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.496357 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:24.674099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.673997 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" event={"ID":"43ec384c-4df2-4802-a5a1-025d107be768","Type":"ContainerStarted","Data":"73cb0a890b936e86ed0a61dccb62ab79394bd4cd14441ce544719378ac1382d3"} Oct 29 22:00:24.674099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.674054 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" event={"ID":"43ec384c-4df2-4802-a5a1-025d107be768","Type":"ContainerStarted","Data":"02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890"} Oct 29 22:00:24.719884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.719843 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" podStartSLOduration=1.719829491 podStartE2EDuration="1.719829491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:24.71789092 +0000 UTC m=+1086.459821624" watchObservedRunningTime="2026-10-29 22:00:24.719829491 +0000 UTC m=+1086.461760163" Oct 29 22:00:24.846146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.844545 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:24.846146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.844697 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:24.846146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.844795 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:24.848428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:24.848395 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:00:25.213546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.213518 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:25.214292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.214217 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:25.215136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.215117 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:00:25.688612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.688525 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" event={"ID":"2e2e0c20-9efb-487c-ac56-72dac20dd253","Type":"ContainerStarted","Data":"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d"} Oct 29 22:00:25.688612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.688589 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" event={"ID":"2e2e0c20-9efb-487c-ac56-72dac20dd253","Type":"ContainerStarted","Data":"d2254275a8d95bda30b4d8625780b76ce573b3a4b5d26e580578c018f4203ab9"} Oct 29 22:00:25.692116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.691365 4543 generic.go:356] "Generic (PLEG): container finished" podID="43ec384c-4df2-4802-a5a1-025d107be768" containerID="73cb0a890b936e86ed0a61dccb62ab79394bd4cd14441ce544719378ac1382d3" exitCode=0 Oct 29 22:00:25.692116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.691483 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" event={"ID":"43ec384c-4df2-4802-a5a1-025d107be768","Type":"ContainerDied","Data":"73cb0a890b936e86ed0a61dccb62ab79394bd4cd14441ce544719378ac1382d3"} Oct 29 22:00:25.795062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:25.795010 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podStartSLOduration=1.794997458 podStartE2EDuration="1.794997458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:25.749452429 +0000 UTC m=+1087.491383104" watchObservedRunningTime="2026-10-29 22:00:25.794997458 +0000 UTC m=+1087.536928132" Oct 29 22:00:26.159108 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:26.159059 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:26.496525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.496500 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:26.497927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.497904 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:26.498012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.497957 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:26.794746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.794713 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr"] Oct 29 22:00:26.803414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.803386 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:26.825379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.825352 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr"] Oct 29 22:00:26.962722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.962700 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:26.964106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.964061 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlx9\" (UniqueName: \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:26.964209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.964178 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:26.964316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:26.964295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.065120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.064999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\") pod \"43ec384c-4df2-4802-a5a1-025d107be768\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " Oct 29 22:00:27.065296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.065260 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\" (UniqueName: \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\") pod \"43ec384c-4df2-4802-a5a1-025d107be768\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " Oct 29 22:00:27.065406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.065335 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\") pod \"43ec384c-4df2-4802-a5a1-025d107be768\" (UID: \"43ec384c-4df2-4802-a5a1-025d107be768\") " Oct 29 22:00:27.065800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.065776 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlx9\" (UniqueName: \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.065887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.065871 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.065995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.065980 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.066666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.066642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.067605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.067577 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca" pod "43ec384c-4df2-4802-a5a1-025d107be768" (UID: "43ec384c-4df2-4802-a5a1-025d107be768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:27.069627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.069602 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca" pod "43ec384c-4df2-4802-a5a1-025d107be768" (UID: "43ec384c-4df2-4802-a5a1-025d107be768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:00:27.070584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.070560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.071047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.071027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk" pod "43ec384c-4df2-4802-a5a1-025d107be768" (UID: "43ec384c-4df2-4802-a5a1-025d107be768"). InnerVolumeSpecName "kube-api-access-xjknk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:27.093437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.093414 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlx9\" (UniqueName: \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\") pod \"maas-api-revoke-sub-1e129c5a-h9vlr\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.167580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.167554 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjknk\" (UniqueName: \"kubernetes.io/projected/43ec384c-4df2-4802-a5a1-025d107be768-kube-api-access-xjknk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:27.167580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.167580 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:27.167754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.167590 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ec384c-4df2-4802-a5a1-025d107be768-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:27.179057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.179036 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:27.497715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.497678 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:27.497907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.497741 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:27.526239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.526211 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr"] Oct 29 22:00:27.526419 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:27.526396 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd88d61_d8a2_4e4c_8f3a_746a5eacb7bc.slice/crio-c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906 WatchSource:0}: Error finding container c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906: Status 404 returned error can't find the container with id c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906 Oct 29 22:00:27.709868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.709833 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" event={"ID":"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc","Type":"ContainerStarted","Data":"c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906"} Oct 29 22:00:27.711321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.711298 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" Oct 29 22:00:27.711321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.711312 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc" event={"ID":"43ec384c-4df2-4802-a5a1-025d107be768","Type":"ContainerDied","Data":"02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890"} Oct 29 22:00:27.711524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:27.711351 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02be9582c0d64e84f88ebcd55384e74965e9a93971083f5f3a873136fc05b890" Oct 29 22:00:27.981391 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:27.981361 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:00:28.496919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:28.496887 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:28.497135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:28.496934 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:28.721636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:28.721607 4543 generic.go:356] "Generic (PLEG): container finished" podID="add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" containerID="bb9c58eedc02af2d0883fa0ef11e48893b77dbd6830ad484bce82c5700cbb986" exitCode=0 Oct 29 22:00:28.721767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:28.721698 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" event={"ID":"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc","Type":"ContainerDied","Data":"bb9c58eedc02af2d0883fa0ef11e48893b77dbd6830ad484bce82c5700cbb986"} Oct 29 22:00:29.505837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:29.505488 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:29.505837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:29.505545 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:30.012137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.012044 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:30.125614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.125583 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\" (UniqueName: \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\") pod \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " Oct 29 22:00:30.125771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.125684 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\") pod \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " Oct 29 22:00:30.125921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.125900 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\") pod \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\" (UID: \"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc\") " Oct 29 22:00:30.126460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.126423 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca" pod "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" (UID: "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:30.126822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.126796 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:30.145111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.128114 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca" pod "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" (UID: "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:00:30.145111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.128221 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9" pod "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" (UID: "add88d61-d8a2-4e4c-8f3a-746a5eacb7bc"). InnerVolumeSpecName "kube-api-access-mwlx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:30.227869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.227839 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlx9\" (UniqueName: \"kubernetes.io/projected/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-kube-api-access-mwlx9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:30.227869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.227869 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:30.497353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.497322 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:30.497514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.497385 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:30.774505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.774465 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" event={"ID":"add88d61-d8a2-4e4c-8f3a-746a5eacb7bc","Type":"ContainerDied","Data":"c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906"} Oct 29 22:00:30.774876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.774521 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48af4524c2e1a83d06fdc1f2208f6fc10a999b20c19ca676058a0c8d1338906" Oct 29 22:00:30.774876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:30.774578 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr" Oct 29 22:00:31.497417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:31.497379 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:00:31.497630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:31.497441 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 29 22:00:32.500851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:32.500823 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:32.501353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:32.501328 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:32.502122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:32.502064 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:00:35.047770 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:35.047735 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:35.102856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.102825 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:00:35.103578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103563 4543 cpu_manager.go:414] "containerMap: removing container" podUID="add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" containerName="revoke-keys" Oct 29 22:00:35.103627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103580 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" containerName="revoke-keys" Oct 29 22:00:35.103627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103607 4543 cpu_manager.go:414] "containerMap: removing container" podUID="43ec384c-4df2-4802-a5a1-025d107be768" containerName="revoke-keys" Oct 29 22:00:35.103627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103612 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ec384c-4df2-4802-a5a1-025d107be768" containerName="revoke-keys" Oct 29 22:00:35.103727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103718 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" containerName="revoke-keys" Oct 29 22:00:35.103763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.103735 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ec384c-4df2-4802-a5a1-025d107be768" containerName="revoke-keys" Oct 29 22:00:35.107647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.107630 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.129725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.129705 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-579f09-dockercfg-h9bc2\"" Oct 29 22:00:35.133543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.133517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljcj\" (UniqueName: \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.133641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.133630 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.134791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.134771 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:00:35.176833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.176804 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-579f09\"" Oct 29 22:00:35.235468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.235420 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:00:35.237401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.237330 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljcj\" (UniqueName: \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.237546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.237427 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.238638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.238605 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.242147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.242123 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.246814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.246790 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:00:35.278141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.266031 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljcj\" (UniqueName: \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\") pod \"payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.424520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.424425 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:35.452148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.452112 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhkm\" (UniqueName: \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.453670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.453649 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.558317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.558276 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhkm\" (UniqueName: \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.558536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.558460 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.559550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.559523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.588337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.588312 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhkm\" (UniqueName: \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\") pod \"payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:35.771265 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:35.771230 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd752ac0_8178_4e74_8e58_a8842d6909d5.slice/crio-426208f582646cb700a0e23185774267a6aa27c3eef4faa0b98f3264bbf6250e WatchSource:0}: Error finding container 426208f582646cb700a0e23185774267a6aa27c3eef4faa0b98f3264bbf6250e: Status 404 returned error can't find the container with id 426208f582646cb700a0e23185774267a6aa27c3eef4faa0b98f3264bbf6250e Oct 29 22:00:35.771858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.771836 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:00:35.823543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.823513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" event={"ID":"cd752ac0-8178-4e74-8e58-a8842d6909d5","Type":"ContainerStarted","Data":"426208f582646cb700a0e23185774267a6aa27c3eef4faa0b98f3264bbf6250e"} Oct 29 22:00:35.882110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:35.882089 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:36.035177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.035102 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs"] Oct 29 22:00:36.040052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.040026 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5e7470-fx5cs"] Oct 29 22:00:36.175013 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:36.174987 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:36.199293 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:36.199258 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816c43f4_5339_4fc1_92d4_9f15f53d5ac2.slice/crio-9d4804cdfc6f263a2434d12367f9f7ebab9f293247e9043ab315abc2bab8c042 WatchSource:0}: Error finding container 9d4804cdfc6f263a2434d12367f9f7ebab9f293247e9043ab315abc2bab8c042: Status 404 returned error can't find the container with id 9d4804cdfc6f263a2434d12367f9f7ebab9f293247e9043ab315abc2bab8c042 Oct 29 22:00:36.199293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.199280 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:00:36.387802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.387739 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86c0885-9d66-4daf-ac04-327ad505cf57" path="/var/lib/kubelet/pods/d86c0885-9d66-4daf-ac04-327ad505cf57/volumes" Oct 29 22:00:36.834005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.833974 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" event={"ID":"816c43f4-5339-4fc1-92d4-9f15f53d5ac2","Type":"ContainerStarted","Data":"6af8eedd65e0764d9bcd17ba3d032c492e36ee072d99975c1541b7735507b2aa"} Oct 29 22:00:36.834219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.834026 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" event={"ID":"816c43f4-5339-4fc1-92d4-9f15f53d5ac2","Type":"ContainerStarted","Data":"9d4804cdfc6f263a2434d12367f9f7ebab9f293247e9043ab315abc2bab8c042"} Oct 29 22:00:36.834219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.834064 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:36.835906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.835881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" event={"ID":"cd752ac0-8178-4e74-8e58-a8842d6909d5","Type":"ContainerStarted","Data":"002e23bfa8761694cb858b785f8d2107e83dacc162900dc7116cac3c71d57b63"} Oct 29 22:00:36.836029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.835927 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:36.896511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.896472 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" podStartSLOduration=1.8964587480000001 podStartE2EDuration="1.896458748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:36.878537054 +0000 UTC m=+1098.620467732" watchObservedRunningTime="2026-10-29 22:00:36.896458748 +0000 UTC m=+1098.638389421" Oct 29 22:00:36.931610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:36.931552 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" podStartSLOduration=1.931535448 podStartE2EDuration="1.931535448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:36.923971651 +0000 UTC m=+1098.665902324" watchObservedRunningTime="2026-10-29 22:00:36.931535448 +0000 UTC m=+1098.673466123" Oct 29 22:00:38.439008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.438976 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx"] Oct 29 22:00:38.444233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.444214 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.463313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.463292 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx"] Oct 29 22:00:38.472116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.470289 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:00:38.472116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.470304 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:00:38.472116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.470373 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:00:38.474350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.474320 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:00:38.499465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.499430 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.499579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.499498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.499649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.499610 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4566s\" (UniqueName: \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.601221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.601201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.601342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.601259 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.601407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.601366 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4566s\" (UniqueName: \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.601857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.601835 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.603688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.603667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.627457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.627434 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4566s\" (UniqueName: \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\") pod \"maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:38.763168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:38.763143 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:39.075362 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:00:39.075332 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f63dcd_4369_42e2_985e_3175d0a90d10.slice/crio-4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526 WatchSource:0}: Error finding container 4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526: Status 404 returned error can't find the container with id 4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526 Oct 29 22:00:39.076134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:39.076108 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx"] Oct 29 22:00:39.869088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:39.869037 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" event={"ID":"16f63dcd-4369-42e2-985e-3175d0a90d10","Type":"ContainerStarted","Data":"ea295ba511d74828e60e93d5de36f9dec398b4d90251145259c5ab8ee22e76e6"} Oct 29 22:00:39.869486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:39.869113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" event={"ID":"16f63dcd-4369-42e2-985e-3175d0a90d10","Type":"ContainerStarted","Data":"4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526"} Oct 29 22:00:39.916997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:39.916953 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" podStartSLOduration=1.916929975 podStartE2EDuration="1.916929975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:39.912490232 +0000 UTC m=+1101.654420906" watchObservedRunningTime="2026-10-29 22:00:39.916929975 +0000 UTC m=+1101.658860648" Oct 29 22:00:40.915855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:40.914418 4543 generic.go:356] "Generic (PLEG): container finished" podID="16f63dcd-4369-42e2-985e-3175d0a90d10" containerID="ea295ba511d74828e60e93d5de36f9dec398b4d90251145259c5ab8ee22e76e6" exitCode=0 Oct 29 22:00:40.915855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:40.914711 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" event={"ID":"16f63dcd-4369-42e2-985e-3175d0a90d10","Type":"ContainerDied","Data":"ea295ba511d74828e60e93d5de36f9dec398b4d90251145259c5ab8ee22e76e6"} Oct 29 22:00:42.600451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.600382 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:42.700644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.700595 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\" (UniqueName: \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\") pod \"16f63dcd-4369-42e2-985e-3175d0a90d10\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " Oct 29 22:00:42.702942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.702918 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\") pod \"16f63dcd-4369-42e2-985e-3175d0a90d10\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " Oct 29 22:00:42.703635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.703518 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca" pod "16f63dcd-4369-42e2-985e-3175d0a90d10" (UID: "16f63dcd-4369-42e2-985e-3175d0a90d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:42.703977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.703956 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\") pod \"16f63dcd-4369-42e2-985e-3175d0a90d10\" (UID: \"16f63dcd-4369-42e2-985e-3175d0a90d10\") " Oct 29 22:00:42.705320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.705284 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:42.743301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.734875 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca" pod "16f63dcd-4369-42e2-985e-3175d0a90d10" (UID: "16f63dcd-4369-42e2-985e-3175d0a90d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:00:42.825154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.818742 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f63dcd-4369-42e2-985e-3175d0a90d10-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:42.874811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.874647 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s" pod "16f63dcd-4369-42e2-985e-3175d0a90d10" (UID: "16f63dcd-4369-42e2-985e-3175d0a90d10"). InnerVolumeSpecName "kube-api-access-4566s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:42.922100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:42.921907 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4566s\" (UniqueName: \"kubernetes.io/projected/16f63dcd-4369-42e2-985e-3175d0a90d10-kube-api-access-4566s\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:43.116134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:43.115853 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" event={"ID":"16f63dcd-4369-42e2-985e-3175d0a90d10","Type":"ContainerDied","Data":"4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526"} Oct 29 22:00:43.116134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:43.115917 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d48199f0a0574166ad956ec4d30d84fbd6c4809a8b5d0f0e2f8f21f99c4f526" Oct 29 22:00:43.116134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:43.115996 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx" Oct 29 22:00:43.875028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:43.873955 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:00:43.942112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:43.938160 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:00:44.168464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.168361 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:00:44.282059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.282018 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.395105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.393385 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.429158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.413156 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.454247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.454159 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:00:44.590104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.580993 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.648379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.647181 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.831102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.830175 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:00:45.028118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:44.979226 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.057101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.029430 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:00:45.184345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.184233 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.185344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.185320 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:00:45.317272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.317208 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.047102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.321165 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.421647 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.600724 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.810508 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.821029 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.845531 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.852681 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:45.919870 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.018763 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.133452 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.178256 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.203306 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:00:46.548690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.548629 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:46.549017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.548968 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.940433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.940395 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.059543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:46.978289 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dnk78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:47.064689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.064651 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.072193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.034152 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:47.072390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.072361 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.593942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.593901 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.713635 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:47.713604 4543 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.009s" Oct 29 22:00:47.718527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.718497 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:00:47.718953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.718939 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:00:47.719704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.719681 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:00:47.720325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.720309 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:47.743185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743161 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:00:47.743355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743342 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:00:47.743498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743487 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:00:47.743613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743603 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:00:47.743708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743698 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:00:47.743809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743800 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:00:47.743919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.743907 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:00:47.821623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.820202 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:00:47.821623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.820975 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll"] Oct 29 22:00:47.863153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.862686 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.924537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.875201 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.924537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.895221 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.929767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.928589 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.955036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.941514 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:47.957368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:47.957344 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:00:48.086126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.086094 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:00:48.260556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.260175 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.281523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.281485 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:00:48.283425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.283385 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:00:48.454103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.429023 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.454394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.439591 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.588380 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:48.588286 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:48.604414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.604373 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.606782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.606749 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:00:48.642879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.642837 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670321 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670373 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670395 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b493e1-92tll"] Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670444 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670473 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:00:48.671136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:48.670503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:00:49.256604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:49.256565 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 22:00:49.358391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:49.358357 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 22:00:50.006845 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:50.006801 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:50.389695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:50.389613 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b99714e-680c-4eac-905d-5cfa16915d86" path="/var/lib/kubelet/pods/8b99714e-680c-4eac-905d-5cfa16915d86/volumes" Oct 29 22:00:54.257304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.257265 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" podUID="fa957374-cdee-4db9-b4ac-69823f5c28c1" containerName="payload-processing" containerID="cri-o://73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415" gracePeriod=25 Oct 29 22:00:54.358928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.358858 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" podUID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" containerName="payload-pre-processing" containerID="cri-o://677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703" gracePeriod=25 Oct 29 22:00:54.606832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.606803 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 22:00:54.655470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.654871 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\" (UniqueName: \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\") pod \"fa957374-cdee-4db9-b4ac-69823f5c28c1\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " Oct 29 22:00:54.655470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.654987 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\") pod \"fa957374-cdee-4db9-b4ac-69823f5c28c1\" (UID: \"fa957374-cdee-4db9-b4ac-69823f5c28c1\") " Oct 29 22:00:54.656193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.656157 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume" pod "fa957374-cdee-4db9-b4ac-69823f5c28c1" (UID: "fa957374-cdee-4db9-b4ac-69823f5c28c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:54.657183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.657108 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa957374-cdee-4db9-b4ac-69823f5c28c1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:54.660852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.658169 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm" pod "fa957374-cdee-4db9-b4ac-69823f5c28c1" (UID: "fa957374-cdee-4db9-b4ac-69823f5c28c1"). InnerVolumeSpecName "kube-api-access-q6fbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:54.683994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.683974 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 22:00:54.757991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.757958 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\" (UniqueName: \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\") pod \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " Oct 29 22:00:54.758180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.757999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\") pod \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\" (UID: \"b05722c4-32ff-465e-abcf-86c2d2f1a42f\") " Oct 29 22:00:54.758489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.758442 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume" pod "b05722c4-32ff-465e-abcf-86c2d2f1a42f" (UID: "b05722c4-32ff-465e-abcf-86c2d2f1a42f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:54.759110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.759059 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6fbm\" (UniqueName: \"kubernetes.io/projected/fa957374-cdee-4db9-b4ac-69823f5c28c1-kube-api-access-q6fbm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:54.759193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.759116 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05722c4-32ff-465e-abcf-86c2d2f1a42f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:54.760469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.760444 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk" pod "b05722c4-32ff-465e-abcf-86c2d2f1a42f" (UID: "b05722c4-32ff-465e-abcf-86c2d2f1a42f"). InnerVolumeSpecName "kube-api-access-nsqvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:54.861361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.861307 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqvk\" (UniqueName: \"kubernetes.io/projected/b05722c4-32ff-465e-abcf-86c2d2f1a42f-kube-api-access-nsqvk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:00:54.934062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.934038 4543 generic.go:356] "Generic (PLEG): container finished" podID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" containerID="677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703" exitCode=0 Oct 29 22:00:54.934229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.934130 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" event={"ID":"b05722c4-32ff-465e-abcf-86c2d2f1a42f","Type":"ContainerDied","Data":"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703"} Oct 29 22:00:54.934229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.934205 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" event={"ID":"b05722c4-32ff-465e-abcf-86c2d2f1a42f","Type":"ContainerDied","Data":"de155fb2ea6ed0ed3a5710fff1214196aab41fe51ea5c37fa6c1236385c827d2"} Oct 29 22:00:54.934229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.934111 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp" Oct 29 22:00:54.934229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.934222 4543 scope.go:122] "RemoveContainer" containerID="677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703" Oct 29 22:00:54.936004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.935985 4543 generic.go:356] "Generic (PLEG): container finished" podID="fa957374-cdee-4db9-b4ac-69823f5c28c1" containerID="73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415" exitCode=0 Oct 29 22:00:54.936124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.936044 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" Oct 29 22:00:54.936124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.936092 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" event={"ID":"fa957374-cdee-4db9-b4ac-69823f5c28c1","Type":"ContainerDied","Data":"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415"} Oct 29 22:00:54.936247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.936151 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r" event={"ID":"fa957374-cdee-4db9-b4ac-69823f5c28c1","Type":"ContainerDied","Data":"3c1552f5163622d8e8c6082dd7a8cc5f5ebba8e145b8bd38abe06b4e2e178ec7"} Oct 29 22:00:54.937891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.937872 4543 generic.go:356] "Generic (PLEG): container finished" podID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerID="b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09" exitCode=0 Oct 29 22:00:54.937996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.937958 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerDied","Data":"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09"} Oct 29 22:00:54.938428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.938410 4543 scope.go:122] "RemoveContainer" containerID="b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09" Oct 29 22:00:54.940135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.940103 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:54.940217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.940193 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:54.940259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.940245 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:00:54.947068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.947051 4543 scope.go:122] "RemoveContainer" containerID="677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703" Oct 29 22:00:54.947350 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:54.947333 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703\": container with ID starting with 677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703 not found: ID does not exist" containerID="677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703" Oct 29 22:00:54.947398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.947357 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703"} err="failed to get container status \"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703\": rpc error: code = NotFound desc = could not find container \"677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703\": container with ID starting with 677e3516750cb5bd74d1a7016e7e57340439e0d184526598c14e234ab4eab703 not found: ID does not exist" Oct 29 22:00:54.947398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.947371 4543 scope.go:122] "RemoveContainer" containerID="73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415" Oct 29 22:00:54.960166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.960146 4543 scope.go:122] "RemoveContainer" containerID="73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415" Oct 29 22:00:54.960413 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:54.960392 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415\": container with ID starting with 73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415 not found: ID does not exist" containerID="73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415" Oct 29 22:00:54.960480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.960420 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415"} err="failed to get container status \"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415\": rpc error: code = NotFound desc = could not find container \"73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415\": container with ID starting with 73b4fbb8923a169feb30357cd8df8f6fbc17f3e2a69d950e0d3d5ab7f0e00415 not found: ID does not exist" Oct 29 22:00:54.995146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:54.995127 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 22:00:55.000247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:55.000227 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a116a2-5d68d7c4f5-jzlxp"] Oct 29 22:00:55.038052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:55.038029 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 22:00:55.047202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:55.047178 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a116a2-865b999574-6j58r"] Oct 29 22:00:55.962540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:55.962504 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerStarted","Data":"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40"} Oct 29 22:00:56.085518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:56.085490 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:00:56.086967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:56.086944 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:00:56.087116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:56.086998 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:00:56.386786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:56.386707 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" path="/var/lib/kubelet/pods/b05722c4-32ff-465e-abcf-86c2d2f1a42f/volumes" Oct 29 22:00:56.387178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:56.387163 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa957374-cdee-4db9-b4ac-69823f5c28c1" path="/var/lib/kubelet/pods/fa957374-cdee-4db9-b4ac-69823f5c28c1/volumes" Oct 29 22:00:57.089495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:57.089459 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:00:57.090001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:57.089522 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:00:58.010688 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:58.010647 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:00:58.085345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:58.085312 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:00:58.085510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:58.085363 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:00:58.689025 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:00:58.688993 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:00:59.085489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:59.085454 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:00:59.085676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:59.085516 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:00:59.551966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:00:59.550891 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:01:00.084955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:00.084910 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:01:00.085367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:00.084980 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:01:01.084950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:01.084911 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 29 22:01:01.085145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:01.084982 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 29 22:01:02.089448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:02.089419 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:01:02.090405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:02.090385 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:01:05.049505 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:05.049472 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:01:08.705882 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:08.705848 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20113d62_44cf_41ff_af86_a9a86e5dad94.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:01:15.036460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:15.036426 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt"] Oct 29 22:01:15.051821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:15.051791 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b1a519-p5xkt"] Oct 29 22:01:16.036358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:16.036328 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns"] Oct 29 22:01:16.049148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:16.049122 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3c40f-7f6ns"] Oct 29 22:01:16.386452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:16.386386 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3bc952-bbf9-4c61-b83f-cf4d65acf471" path="/var/lib/kubelet/pods/8f3bc952-bbf9-4c61-b83f-cf4d65acf471/volumes" Oct 29 22:01:16.386848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:16.386832 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f364c227-13d2-4b69-92a1-854614113206" path="/var/lib/kubelet/pods/f364c227-13d2-4b69-92a1-854614113206/volumes" Oct 29 22:01:17.351310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.351283 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:01:17.352248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352227 4543 cpu_manager.go:414] "containerMap: removing container" podUID="16f63dcd-4369-42e2-985e-3175d0a90d10" containerName="revoke-keys" Oct 29 22:01:17.352248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352249 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f63dcd-4369-42e2-985e-3175d0a90d10" containerName="revoke-keys" Oct 29 22:01:17.352441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352262 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fa957374-cdee-4db9-b4ac-69823f5c28c1" containerName="payload-processing" Oct 29 22:01:17.352441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352268 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa957374-cdee-4db9-b4ac-69823f5c28c1" containerName="payload-processing" Oct 29 22:01:17.352441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352293 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" containerName="payload-pre-processing" Oct 29 22:01:17.352441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352301 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" containerName="payload-pre-processing" Oct 29 22:01:17.352632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352497 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f63dcd-4369-42e2-985e-3175d0a90d10" containerName="revoke-keys" Oct 29 22:01:17.352632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352510 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa957374-cdee-4db9-b4ac-69823f5c28c1" containerName="payload-processing" Oct 29 22:01:17.352632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.352523 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05722c4-32ff-465e-abcf-86c2d2f1a42f" containerName="payload-pre-processing" Oct 29 22:01:17.356359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.356339 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.382417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.382395 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:01:17.498586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.498547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.498777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.498613 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.498901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.498865 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x46b\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.498944 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.499040 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.499109 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.499193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.499234 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.499319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.499302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601139 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x46b\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601184 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601233 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601335 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601662 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601688 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601722 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.601902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.601817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.602294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.602270 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.602460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.602297 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.604315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.604294 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.604418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.604386 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.647583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.642331 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.647583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.642615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x46b\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:17.671927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:17.671900 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:18.005880 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:18.005849 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ade5319_70bd_458b_a7ff_667141033adf.slice/crio-a28da471d9f1376118be718d051d6e7a59badc1c6dcd49686ee478124c821b7a WatchSource:0}: Error finding container a28da471d9f1376118be718d051d6e7a59badc1c6dcd49686ee478124c821b7a: Status 404 returned error can't find the container with id a28da471d9f1376118be718d051d6e7a59badc1c6dcd49686ee478124c821b7a Oct 29 22:01:18.006371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.006348 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:01:18.008504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.008473 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:18.008579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.008537 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:18.008579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.008564 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:18.175476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.175422 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" event={"ID":"7ade5319-70bd-458b-a7ff-667141033adf","Type":"ContainerStarted","Data":"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02"} Oct 29 22:01:18.175652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.175499 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" event={"ID":"7ade5319-70bd-458b-a7ff-667141033adf","Type":"ContainerStarted","Data":"a28da471d9f1376118be718d051d6e7a59badc1c6dcd49686ee478124c821b7a"} Oct 29 22:01:18.227730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.227665 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podStartSLOduration=1.227647387 podStartE2EDuration="1.227647387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:18.223724877 +0000 UTC m=+1139.965655553" watchObservedRunningTime="2026-10-29 22:01:18.227647387 +0000 UTC m=+1139.969578063" Oct 29 22:01:18.672878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.672840 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:18.808677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.808647 4543 scope.go:122] "RemoveContainer" containerID="e3bc1e71c448ae7a289b2a359f78880fd24781c4b487e7d1bc8669c56e3549ee" Oct 29 22:01:18.819997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.819980 4543 scope.go:122] "RemoveContainer" containerID="544781a9d08c441064d29cc66a2cb40afe505e17830388ec19b6f5e8cb343c92" Oct 29 22:01:18.861709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.861676 4543 scope.go:122] "RemoveContainer" containerID="2be410dd74119a14ed39cd832dfa78b12356b2aeb4da4507afc3eda9ece26f0f" Oct 29 22:01:18.875468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:18.875441 4543 scope.go:122] "RemoveContainer" containerID="6b0974f4c46553028d0c7137db40a3985410bd42991a02018a81709b7211f10e" Oct 29 22:01:19.711146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:19.708624 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:19.711146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:19.708688 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:20.675101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:20.674482 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:20.675101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:20.674549 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:21.678102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:21.672416 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:21.678102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:21.672474 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:22.675102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:22.672830 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:22.675102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:22.672890 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:23.672916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:23.672883 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:23.673315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:23.672933 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:24.673019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:24.672985 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 29 22:01:24.673398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:24.673035 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 29 22:01:25.047102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.047045 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf"] Oct 29 22:01:25.052183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.052166 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.071439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.071414 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:01:25.072281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.072252 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf"] Oct 29 22:01:25.078668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.078644 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:01:25.088290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.088269 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:01:25.088414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.088348 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:01:25.130672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.130647 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.130819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.130684 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tqk\" (UniqueName: \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.130819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.130812 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.232526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.232498 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.232619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.232551 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tqk\" (UniqueName: \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.232668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.232651 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.233180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.233159 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.235181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.235160 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.258952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.258929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tqk\" (UniqueName: \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\") pod \"maas-api-revoke-keys-e2e-api-a-d4082f-fgspf\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.367425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.367328 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:25.702757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:25.702723 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf"] Oct 29 22:01:25.703215 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:25.702882 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a8a2b3_d7c7_48d5_803a_e149a9019f6c.slice/crio-fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4 WatchSource:0}: Error finding container fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4: Status 404 returned error can't find the container with id fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4 Oct 29 22:01:26.395127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.395006 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" event={"ID":"08a8a2b3-d7c7-48d5-803a-e149a9019f6c","Type":"ContainerStarted","Data":"3d74adf9be66d9890a79fdecc51639514706b13a6777b51edfa9600f120c4d0b"} Oct 29 22:01:26.395127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.395093 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" event={"ID":"08a8a2b3-d7c7-48d5-803a-e149a9019f6c","Type":"ContainerStarted","Data":"fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4"} Oct 29 22:01:26.440377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.440329 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" podStartSLOduration=1.4403125399999999 podStartE2EDuration="1.44031254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:26.439743308 +0000 UTC m=+1148.181673982" watchObservedRunningTime="2026-10-29 22:01:26.44031254 +0000 UTC m=+1148.182243229" Oct 29 22:01:26.714203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.714177 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:01:26.714586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.714229 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:26.970695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.970613 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:26.971155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.971131 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:26.971826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:26.971806 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:01:27.404490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.404414 4543 generic.go:356] "Generic (PLEG): container finished" podID="08a8a2b3-d7c7-48d5-803a-e149a9019f6c" containerID="3d74adf9be66d9890a79fdecc51639514706b13a6777b51edfa9600f120c4d0b" exitCode=0 Oct 29 22:01:27.404621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.404494 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" event={"ID":"08a8a2b3-d7c7-48d5-803a-e149a9019f6c","Type":"ContainerDied","Data":"3d74adf9be66d9890a79fdecc51639514706b13a6777b51edfa9600f120c4d0b"} Oct 29 22:01:27.828177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.828145 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:01:27.832197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.832179 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.851459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.851416 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-76c786-openshift-default-dockercfg-msjfs\"" Oct 29 22:01:27.858301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.858247 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:01:27.880693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.880662 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.880923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.880902 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.880987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.880939 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qng\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881085 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881163 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881310 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881360 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.881634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.881517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.961918 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:27.961887 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:01:27.982812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.982778 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.982961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.982944 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.982975 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qng\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983011 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983099 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983211 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983248 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983289 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983323 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983338 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983528 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983791 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.983912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.983894 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.985178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.985148 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:27.985861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:27.985839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:28.057294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.047910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qng\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:28.068098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.063554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\") pod \"e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:28.164153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.157403 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:28.542969 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:28.542941 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81029b9d_4392_4638_ad49_600ac5dd1996.slice/crio-1d279201bc127be86868db7590440c5793ff255c1e2474c58a2af299b3324a76 WatchSource:0}: Error finding container 1d279201bc127be86868db7590440c5793ff255c1e2474c58a2af299b3324a76: Status 404 returned error can't find the container with id 1d279201bc127be86868db7590440c5793ff255c1e2474c58a2af299b3324a76 Oct 29 22:01:28.546614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.546573 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:28.546732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.546663 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:28.546732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.546710 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:01:28.547350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.547279 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:01:28.556329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.556308 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:28.621660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.621219 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\" (UniqueName: \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\") pod \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " Oct 29 22:01:28.621660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.621493 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\") pod \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " Oct 29 22:01:28.621660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.621543 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\") pod \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\" (UID: \"08a8a2b3-d7c7-48d5-803a-e149a9019f6c\") " Oct 29 22:01:28.622565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.622539 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca" pod "08a8a2b3-d7c7-48d5-803a-e149a9019f6c" (UID: "08a8a2b3-d7c7-48d5-803a-e149a9019f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:28.640416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.640390 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk" pod "08a8a2b3-d7c7-48d5-803a-e149a9019f6c" (UID: "08a8a2b3-d7c7-48d5-803a-e149a9019f6c"). InnerVolumeSpecName "kube-api-access-z7tqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:28.643570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.642754 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca" pod "08a8a2b3-d7c7-48d5-803a-e149a9019f6c" (UID: "08a8a2b3-d7c7-48d5-803a-e149a9019f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:01:28.724152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.724061 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:28.724152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.724109 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:28.724152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:28.724122 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7tqk\" (UniqueName: \"kubernetes.io/projected/08a8a2b3-d7c7-48d5-803a-e149a9019f6c-kube-api-access-z7tqk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:29.422784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.422702 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" Oct 29 22:01:29.423257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.422707 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf" event={"ID":"08a8a2b3-d7c7-48d5-803a-e149a9019f6c","Type":"ContainerDied","Data":"fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4"} Oct 29 22:01:29.423257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.422878 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5e95594e6052f3ad4ad47851ed1379a16de5e630a05efbe1e2292fb7c6afa4" Oct 29 22:01:29.424811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.424785 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerStarted","Data":"2618bc8622c7a9a403dcdfe914d49ef2da301be712628f8d8e3d3b3445ce1bec"} Oct 29 22:01:29.424940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.424826 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerStarted","Data":"1d279201bc127be86868db7590440c5793ff255c1e2474c58a2af299b3324a76"} Oct 29 22:01:29.499022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.498961 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podStartSLOduration=2.498941658 podStartE2EDuration="2.498941658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:29.486364029 +0000 UTC m=+1151.228294736" watchObservedRunningTime="2026-10-29 22:01:29.498941658 +0000 UTC m=+1151.240872331" Oct 29 22:01:29.675438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.675339 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf"] Oct 29 22:01:29.676665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.676637 4543 cpu_manager.go:414] "containerMap: removing container" podUID="08a8a2b3-d7c7-48d5-803a-e149a9019f6c" containerName="revoke-keys" Oct 29 22:01:29.676665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.676662 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a8a2b3-d7c7-48d5-803a-e149a9019f6c" containerName="revoke-keys" Oct 29 22:01:29.680920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.678830 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a8a2b3-d7c7-48d5-803a-e149a9019f6c" containerName="revoke-keys" Oct 29 22:01:29.682758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.682733 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.702979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.702946 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf"] Oct 29 22:01:29.703311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.703292 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:01:29.703856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.703833 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:01:29.703955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.703936 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:01:29.704070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.703972 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:01:29.843337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.843310 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.843512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.843355 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv67l\" (UniqueName: \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.843595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.843573 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.944782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.944699 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.944931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.944828 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.944931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.944862 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv67l\" (UniqueName: \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.945441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.945420 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.947437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.947413 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:29.975532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:29.975512 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv67l\" (UniqueName: \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\") pod \"maas-api-revoke-sub-d19d4826-nfspf\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:30.006248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.006223 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:30.158611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.158580 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:30.163961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.163928 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:30.315940 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:30.315913 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904cc8ad_57db_424e_abea_e35c6ffd3c3d.slice/crio-c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a WatchSource:0}: Error finding container c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a: Status 404 returned error can't find the container with id c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a Oct 29 22:01:30.316245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.316225 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf"] Oct 29 22:01:30.433805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.433777 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" event={"ID":"904cc8ad-57db-424e-abea-e35c6ffd3c3d","Type":"ContainerStarted","Data":"c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a"} Oct 29 22:01:30.434377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.434363 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:30.435249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:30.435233 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:01:31.456786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:31.456754 4543 generic.go:356] "Generic (PLEG): container finished" podID="904cc8ad-57db-424e-abea-e35c6ffd3c3d" containerID="85e8ecfdb1fb945e52952a8adedd564fb4366886adcd5efda782fdb32d878eb7" exitCode=0 Oct 29 22:01:31.457255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:31.456833 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" event={"ID":"904cc8ad-57db-424e-abea-e35c6ffd3c3d","Type":"ContainerDied","Data":"85e8ecfdb1fb945e52952a8adedd564fb4366886adcd5efda782fdb32d878eb7"} Oct 29 22:01:32.652594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.652570 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:32.784908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.784841 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\") pod \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " Oct 29 22:01:32.785257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.785234 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\" (UniqueName: \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\") pod \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " Oct 29 22:01:32.785319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.785256 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca" pod "904cc8ad-57db-424e-abea-e35c6ffd3c3d" (UID: "904cc8ad-57db-424e-abea-e35c6ffd3c3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:32.785319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.785278 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\") pod \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\" (UID: \"904cc8ad-57db-424e-abea-e35c6ffd3c3d\") " Oct 29 22:01:32.785857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.785834 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:32.787371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.787343 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca" pod "904cc8ad-57db-424e-abea-e35c6ffd3c3d" (UID: "904cc8ad-57db-424e-abea-e35c6ffd3c3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:01:32.787447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.787403 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l" pod "904cc8ad-57db-424e-abea-e35c6ffd3c3d" (UID: "904cc8ad-57db-424e-abea-e35c6ffd3c3d"). InnerVolumeSpecName "kube-api-access-xv67l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:32.890166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.888738 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv67l\" (UniqueName: \"kubernetes.io/projected/904cc8ad-57db-424e-abea-e35c6ffd3c3d-kube-api-access-xv67l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:32.890166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:32.888779 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904cc8ad-57db-424e-abea-e35c6ffd3c3d-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:33.474690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:33.474660 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" event={"ID":"904cc8ad-57db-424e-abea-e35c6ffd3c3d","Type":"ContainerDied","Data":"c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a"} Oct 29 22:01:33.474690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:33.474685 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf" Oct 29 22:01:33.474884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:33.474706 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f19bf8ed6a515f1cf4feee099ce8d8e314a07188cac2644d73e8527b25670a" Oct 29 22:01:34.332590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.332558 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:01:34.333333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.333317 4543 cpu_manager.go:414] "containerMap: removing container" podUID="904cc8ad-57db-424e-abea-e35c6ffd3c3d" containerName="revoke-keys" Oct 29 22:01:34.333428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.333334 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904cc8ad-57db-424e-abea-e35c6ffd3c3d" containerName="revoke-keys" Oct 29 22:01:34.333480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.333470 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="904cc8ad-57db-424e-abea-e35c6ffd3c3d" containerName="revoke-keys" Oct 29 22:01:34.337690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.337672 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.368836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.368808 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:01:34.375365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.375330 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76c786\"" Oct 29 22:01:34.406507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.406475 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:01:34.413168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.413144 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.431728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.431703 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:01:34.432468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.432434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.432605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.432583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpbq\" (UniqueName: \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.433900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.433882 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76c786-dockercfg-g8rnm\"" Oct 29 22:01:34.595154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.578497 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.595154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.578577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.595154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.578709 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpbq\" (UniqueName: \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.595154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.578736 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvwk\" (UniqueName: \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.600142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.600061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.677315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.677276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpbq\" (UniqueName: \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\") pod \"payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.694489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.694458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvwk\" (UniqueName: \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.694937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.694919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.695799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.695780 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:34.698172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.698157 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:34.732824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:34.732793 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvwk\" (UniqueName: \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\") pod \"payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:35.025608 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:35.025577 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe46f751_6e61_4767_934d_9df40fc58f5e.slice/crio-357ea9c3a0b1d7006e36dfcaf8aad05c799c0427ef7f2b29779252a2a05b8b24 WatchSource:0}: Error finding container 357ea9c3a0b1d7006e36dfcaf8aad05c799c0427ef7f2b29779252a2a05b8b24: Status 404 returned error can't find the container with id 357ea9c3a0b1d7006e36dfcaf8aad05c799c0427ef7f2b29779252a2a05b8b24 Oct 29 22:01:35.026912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.026892 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:01:35.031616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.031596 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:35.351140 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:35.351043 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f6becd_2509_4dc8_828e_f0a1b00dbc92.slice/crio-d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a WatchSource:0}: Error finding container d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a: Status 404 returned error can't find the container with id d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a Oct 29 22:01:35.352591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.352571 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:01:35.632647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.632297 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" event={"ID":"01f6becd-2509-4dc8-828e-f0a1b00dbc92","Type":"ContainerStarted","Data":"1a1ae2cbe20b4f8f37a9e6676728b170c55de66195b4e13ad7896dfcd5420575"} Oct 29 22:01:35.632647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.632411 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:35.632647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.632434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" event={"ID":"01f6becd-2509-4dc8-828e-f0a1b00dbc92","Type":"ContainerStarted","Data":"d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a"} Oct 29 22:01:35.635978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.634611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" event={"ID":"be46f751-6e61-4767-934d-9df40fc58f5e","Type":"ContainerStarted","Data":"8b1413fcfec30f53f5e41a38f36f52a0daece7a98ccb5a73867fcd8bba8f4cab"} Oct 29 22:01:35.635978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.634673 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" event={"ID":"be46f751-6e61-4767-934d-9df40fc58f5e","Type":"ContainerStarted","Data":"357ea9c3a0b1d7006e36dfcaf8aad05c799c0427ef7f2b29779252a2a05b8b24"} Oct 29 22:01:35.677573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.677516 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" podStartSLOduration=1.677502335 podStartE2EDuration="1.677502335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:35.676678797 +0000 UTC m=+1157.418609470" watchObservedRunningTime="2026-10-29 22:01:35.677502335 +0000 UTC m=+1157.419433002" Oct 29 22:01:35.746580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.746526 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" podStartSLOduration=1.746505985 podStartE2EDuration="1.746505985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:35.725046976 +0000 UTC m=+1157.466977653" watchObservedRunningTime="2026-10-29 22:01:35.746505985 +0000 UTC m=+1157.488436662" Oct 29 22:01:35.886404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.886314 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 22:01:35.965059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:35.965027 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 22:01:36.643811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:36.643785 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:39.556230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.556194 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:01:39.560987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.560968 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.578725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.578703 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:01:39.586244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.586218 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fd295\"" Oct 29 22:01:39.586355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.586227 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fd295-dockercfg-6d86k\"" Oct 29 22:01:39.791475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.791436 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8z8\" (UniqueName: \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.791835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.791813 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.852022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.851933 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:01:39.857886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.857857 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:01:39.858048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.857970 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:39.893402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.893372 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.893525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.893408 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjhb\" (UniqueName: \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:39.893663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.893649 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:39.893721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.893682 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8z8\" (UniqueName: \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.894021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.894002 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.920296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.920274 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8z8\" (UniqueName: \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:39.995244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.995212 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjhb\" (UniqueName: \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:39.995524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.995497 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:39.996309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:39.996288 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:40.021659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.021624 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjhb\" (UniqueName: \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\") pod \"payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:40.179344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.179248 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:40.180870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.180833 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:40.479482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.479446 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx"] Oct 29 22:01:40.479650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.479501 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a116a2-wn8dx"] Oct 29 22:01:40.735228 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:40.735189 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf91cf25_f4d7_4fc2_bbf9_1f857cc33b76.slice/crio-a9b8336a5d03f6c6ee57caf0a440d4bc5ab56cc9a4965b996a55fb76a1e50cd7 WatchSource:0}: Error finding container a9b8336a5d03f6c6ee57caf0a440d4bc5ab56cc9a4965b996a55fb76a1e50cd7: Status 404 returned error can't find the container with id a9b8336a5d03f6c6ee57caf0a440d4bc5ab56cc9a4965b996a55fb76a1e50cd7 Oct 29 22:01:40.735965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.735881 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:01:40.776839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.776806 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:01:40.777068 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:40.777046 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ea458d_3041_41c3_8159_c9838eb1173e.slice/crio-4e2ab2ad1b111f6b5d23300dddd2c90f387d78a7772f6ca6fc04cc0350f2ec3c WatchSource:0}: Error finding container 4e2ab2ad1b111f6b5d23300dddd2c90f387d78a7772f6ca6fc04cc0350f2ec3c: Status 404 returned error can't find the container with id 4e2ab2ad1b111f6b5d23300dddd2c90f387d78a7772f6ca6fc04cc0350f2ec3c Oct 29 22:01:40.799591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.799560 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" event={"ID":"79ea458d-3041-41c3-8159-c9838eb1173e","Type":"ContainerStarted","Data":"4e2ab2ad1b111f6b5d23300dddd2c90f387d78a7772f6ca6fc04cc0350f2ec3c"} Oct 29 22:01:40.801590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.801563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" event={"ID":"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76","Type":"ContainerStarted","Data":"a9b8336a5d03f6c6ee57caf0a440d4bc5ab56cc9a4965b996a55fb76a1e50cd7"} Oct 29 22:01:40.887437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.887409 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" podUID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" containerName="payload-processing" containerID="cri-o://12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a" gracePeriod=25 Oct 29 22:01:40.966038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:40.965961 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" podUID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" containerName="payload-pre-processing" containerID="cri-o://d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c" gracePeriod=25 Oct 29 22:01:41.267661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.267635 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 22:01:41.294292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.294270 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 22:01:41.337255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.337222 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\") pod \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " Oct 29 22:01:41.337418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.337310 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\" (UniqueName: \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\") pod \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " Oct 29 22:01:41.337418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.337398 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\") pod \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\" (UID: \"d22bc9af-ec20-4f68-9907-558c8d2bd0e7\") " Oct 29 22:01:41.337553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.337432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\" (UniqueName: \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\") pod \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\" (UID: \"de3c2678-90bd-4423-9f4f-5d11b0b434ab\") " Oct 29 22:01:41.340949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.339513 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume" pod "d22bc9af-ec20-4f68-9907-558c8d2bd0e7" (UID: "d22bc9af-ec20-4f68-9907-558c8d2bd0e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:41.340949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.339675 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume" pod "de3c2678-90bd-4423-9f4f-5d11b0b434ab" (UID: "de3c2678-90bd-4423-9f4f-5d11b0b434ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:41.342148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.342123 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l" pod "d22bc9af-ec20-4f68-9907-558c8d2bd0e7" (UID: "d22bc9af-ec20-4f68-9907-558c8d2bd0e7"). InnerVolumeSpecName "kube-api-access-brf8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:41.342505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.342480 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt" pod "de3c2678-90bd-4423-9f4f-5d11b0b434ab" (UID: "de3c2678-90bd-4423-9f4f-5d11b0b434ab"). InnerVolumeSpecName "kube-api-access-2s4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:41.440173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.440143 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brf8l\" (UniqueName: \"kubernetes.io/projected/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-kube-api-access-brf8l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:41.440173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.440176 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22bc9af-ec20-4f68-9907-558c8d2bd0e7-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:41.440396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.440191 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4tt\" (UniqueName: \"kubernetes.io/projected/de3c2678-90bd-4423-9f4f-5d11b0b434ab-kube-api-access-2s4tt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:41.440396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.440205 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3c2678-90bd-4423-9f4f-5d11b0b434ab-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:41.814825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.814793 4543 generic.go:356] "Generic (PLEG): container finished" podID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" containerID="12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a" exitCode=0 Oct 29 22:01:41.815360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.814838 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" Oct 29 22:01:41.815360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.814920 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" event={"ID":"de3c2678-90bd-4423-9f4f-5d11b0b434ab","Type":"ContainerDied","Data":"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a"} Oct 29 22:01:41.815360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.814982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p" event={"ID":"de3c2678-90bd-4423-9f4f-5d11b0b434ab","Type":"ContainerDied","Data":"0239fc15e1de49ea902332c5fa521e01827a4130289545b5bbc6e63b5720707c"} Oct 29 22:01:41.815360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.815009 4543 scope.go:122] "RemoveContainer" containerID="12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a" Oct 29 22:01:41.816555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.816531 4543 generic.go:356] "Generic (PLEG): container finished" podID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" containerID="d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c" exitCode=0 Oct 29 22:01:41.816674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.816570 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" Oct 29 22:01:41.816674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.816617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" event={"ID":"d22bc9af-ec20-4f68-9907-558c8d2bd0e7","Type":"ContainerDied","Data":"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c"} Oct 29 22:01:41.816795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.816711 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk" event={"ID":"d22bc9af-ec20-4f68-9907-558c8d2bd0e7","Type":"ContainerDied","Data":"8aced9ec4e52a01c71aa02e3b474b3e819081d9840dae3eefd85b310a7193b5e"} Oct 29 22:01:41.819093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.819052 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" event={"ID":"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76","Type":"ContainerStarted","Data":"3f18b1fbf37c9c1ab0169d9102170a80b0044d5d1fb66b8d9e41b46bf5e3a721"} Oct 29 22:01:41.819201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.819166 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:41.822867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.822727 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" event={"ID":"79ea458d-3041-41c3-8159-c9838eb1173e","Type":"ContainerStarted","Data":"fce248f8e6ad81c084fc643e3648dc8b33a79f8f93ccf112029bd68739edca60"} Oct 29 22:01:41.823536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.823463 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:41.829738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.829719 4543 scope.go:122] "RemoveContainer" containerID="12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a" Oct 29 22:01:41.830020 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:41.830002 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a\": container with ID starting with 12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a not found: ID does not exist" containerID="12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a" Oct 29 22:01:41.831151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.830028 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a"} err="failed to get container status \"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a\": rpc error: code = NotFound desc = could not find container \"12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a\": container with ID starting with 12f3941951f811161e659ea0064bc4054e79d76d3d418a3c9e94a0f4afe6875a not found: ID does not exist" Oct 29 22:01:41.831151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.830046 4543 scope.go:122] "RemoveContainer" containerID="d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c" Oct 29 22:01:41.844221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.844162 4543 scope.go:122] "RemoveContainer" containerID="d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c" Oct 29 22:01:41.844811 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:41.844778 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c\": container with ID starting with d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c not found: ID does not exist" containerID="d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c" Oct 29 22:01:41.844900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.844823 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c"} err="failed to get container status \"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c\": rpc error: code = NotFound desc = could not find container \"d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c\": container with ID starting with d2f3df2cdd53facaa108ddc17228a77e7654f31851eba697811736d2bae00e1c not found: ID does not exist" Oct 29 22:01:41.883816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.883778 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 22:01:41.891434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.891395 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4082f-878696498-6269p"] Oct 29 22:01:41.933957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.933877 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" podStartSLOduration=2.933856728 podStartE2EDuration="2.933856728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:41.92423479 +0000 UTC m=+1163.666165475" watchObservedRunningTime="2026-10-29 22:01:41.933856728 +0000 UTC m=+1163.675787403" Oct 29 22:01:41.975965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.975934 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 22:01:41.982054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:41.982027 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4082f-7f4cfc9d59-w6kzk"] Oct 29 22:01:42.028594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:42.028550 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" podStartSLOduration=3.028536916 podStartE2EDuration="3.028536916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:42.027253103 +0000 UTC m=+1163.769183776" watchObservedRunningTime="2026-10-29 22:01:42.028536916 +0000 UTC m=+1163.770467588" Oct 29 22:01:42.387369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:42.387331 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f63dcd-4369-42e2-985e-3175d0a90d10" path="/var/lib/kubelet/pods/16f63dcd-4369-42e2-985e-3175d0a90d10/volumes" Oct 29 22:01:42.387807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:42.387792 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" path="/var/lib/kubelet/pods/d22bc9af-ec20-4f68-9907-558c8d2bd0e7/volumes" Oct 29 22:01:42.388258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:42.388242 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" path="/var/lib/kubelet/pods/de3c2678-90bd-4423-9f4f-5d11b0b434ab/volumes" Oct 29 22:01:46.648713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:46.648685 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:01:47.262564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.262525 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:01:47.263349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263329 4543 cpu_manager.go:414] "containerMap: removing container" podUID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" containerName="payload-processing" Oct 29 22:01:47.263349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263349 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" containerName="payload-processing" Oct 29 22:01:47.263484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263384 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" containerName="payload-pre-processing" Oct 29 22:01:47.263484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263390 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" containerName="payload-pre-processing" Oct 29 22:01:47.263553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263490 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22bc9af-ec20-4f68-9907-558c8d2bd0e7" containerName="payload-pre-processing" Oct 29 22:01:47.263553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.263505 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3c2678-90bd-4423-9f4f-5d11b0b434ab" containerName="payload-processing" Oct 29 22:01:47.269149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.269131 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.302279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.302256 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:01:47.306324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.306298 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fcce6b\"" Oct 29 22:01:47.306453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.306334 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fcce6b-dockercfg-mvdrv\"" Oct 29 22:01:47.452647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.452607 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:01:47.469473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.469402 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.482278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.482166 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.482444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.482315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w8l\" (UniqueName: \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.536156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.536055 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:01:47.584225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.584187 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.584428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.584272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.584613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.584583 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w8l\" (UniqueName: \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.584776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.584744 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2x8\" (UniqueName: \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.585305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.585265 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.612971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.612929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w8l\" (UniqueName: \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\") pod \"payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:47.655249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.655221 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:01:47.689101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.686296 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.689101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.686469 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2x8\" (UniqueName: \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.689101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.687695 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.748326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.748292 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2x8\" (UniqueName: \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\") pod \"payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.846656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.846569 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:47.884522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:47.884495 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:48.163092 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:48.163028 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d1a98f_7443_491c_bf7b_e50d45255017.slice/crio-8b92bbafa879c253bcbe6578a4059d26a823c253db71cdbe84c2176459fe1a3a WatchSource:0}: Error finding container 8b92bbafa879c253bcbe6578a4059d26a823c253db71cdbe84c2176459fe1a3a: Status 404 returned error can't find the container with id 8b92bbafa879c253bcbe6578a4059d26a823c253db71cdbe84c2176459fe1a3a Oct 29 22:01:48.164715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.164694 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:01:48.229922 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:48.229898 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f36a6c_459b_4d2a_94b5_25e28718f56b.slice/crio-81196075791e59c0af1d140bddab7dc7e38c0a1b78837802366582c3e9583fe4 WatchSource:0}: Error finding container 81196075791e59c0af1d140bddab7dc7e38c0a1b78837802366582c3e9583fe4: Status 404 returned error can't find the container with id 81196075791e59c0af1d140bddab7dc7e38c0a1b78837802366582c3e9583fe4 Oct 29 22:01:48.231199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.231174 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:01:48.952992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.952955 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" event={"ID":"25d1a98f-7443-491c-bf7b-e50d45255017","Type":"ContainerStarted","Data":"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1"} Oct 29 22:01:48.953542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.953039 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" event={"ID":"25d1a98f-7443-491c-bf7b-e50d45255017","Type":"ContainerStarted","Data":"8b92bbafa879c253bcbe6578a4059d26a823c253db71cdbe84c2176459fe1a3a"} Oct 29 22:01:48.953542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.953063 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:01:48.954810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.954777 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" event={"ID":"36f36a6c-459b-4d2a-94b5-25e28718f56b","Type":"ContainerStarted","Data":"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536"} Oct 29 22:01:48.954959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.954840 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:48.954959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:48.954864 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" event={"ID":"36f36a6c-459b-4d2a-94b5-25e28718f56b","Type":"ContainerStarted","Data":"81196075791e59c0af1d140bddab7dc7e38c0a1b78837802366582c3e9583fe4"} Oct 29 22:01:49.003483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.003445 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" podStartSLOduration=2.003433095 podStartE2EDuration="2.003433095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:48.997568706 +0000 UTC m=+1170.739499379" watchObservedRunningTime="2026-10-29 22:01:49.003433095 +0000 UTC m=+1170.745363798" Oct 29 22:01:49.046116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.046049 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" podStartSLOduration=2.046030421 podStartE2EDuration="2.046030421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:49.044917151 +0000 UTC m=+1170.786847836" watchObservedRunningTime="2026-10-29 22:01:49.046030421 +0000 UTC m=+1170.787961094" Oct 29 22:01:49.637466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.637435 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9"] Oct 29 22:01:49.642332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.642310 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.658799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.658775 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9"] Oct 29 22:01:49.665937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.665909 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:01:49.666525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.666505 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:01:49.666643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.666625 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:01:49.673159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.673138 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:01:49.731915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.731886 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.732229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.732208 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.732357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.732340 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kcc\" (UniqueName: \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.833520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.833501 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.833608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.833569 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kcc\" (UniqueName: \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.833608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.833605 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.834335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.834306 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.836273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.836224 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.866112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.866045 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kcc\" (UniqueName: \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\") pod \"maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:49.960784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:49.960704 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:50.282353 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:01:50.282323 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da7a40_b868_485c_9d1f_fd85d3f1a252.slice/crio-9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e WatchSource:0}: Error finding container 9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e: Status 404 returned error can't find the container with id 9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e Oct 29 22:01:50.283487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:50.283468 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9"] Oct 29 22:01:50.975124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:50.975063 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" event={"ID":"79da7a40-b868-485c-9d1f-fd85d3f1a252","Type":"ContainerStarted","Data":"4aff5a0aa1796655bdc5e8d11cc5b6687058d26f37fa14a0abaf42a3b8793ea1"} Oct 29 22:01:50.975539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:50.975154 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" event={"ID":"79da7a40-b868-485c-9d1f-fd85d3f1a252","Type":"ContainerStarted","Data":"9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e"} Oct 29 22:01:51.024686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:51.024636 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" podStartSLOduration=2.024621578 podStartE2EDuration="2.024621578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:51.019555789 +0000 UTC m=+1172.761486462" watchObservedRunningTime="2026-10-29 22:01:51.024621578 +0000 UTC m=+1172.766552251" Oct 29 22:01:51.985413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:51.985380 4543 generic.go:356] "Generic (PLEG): container finished" podID="79da7a40-b868-485c-9d1f-fd85d3f1a252" containerID="4aff5a0aa1796655bdc5e8d11cc5b6687058d26f37fa14a0abaf42a3b8793ea1" exitCode=0 Oct 29 22:01:51.985780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:51.985457 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" event={"ID":"79da7a40-b868-485c-9d1f-fd85d3f1a252","Type":"ContainerDied","Data":"4aff5a0aa1796655bdc5e8d11cc5b6687058d26f37fa14a0abaf42a3b8793ea1"} Oct 29 22:01:52.842070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:52.842039 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:01:53.175160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.175131 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:53.282204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.282178 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\") pod \"79da7a40-b868-485c-9d1f-fd85d3f1a252\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " Oct 29 22:01:53.282382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.282265 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\" (UniqueName: \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\") pod \"79da7a40-b868-485c-9d1f-fd85d3f1a252\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " Oct 29 22:01:53.282479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.282465 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\") pod \"79da7a40-b868-485c-9d1f-fd85d3f1a252\" (UID: \"79da7a40-b868-485c-9d1f-fd85d3f1a252\") " Oct 29 22:01:53.282762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.282739 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca" pod "79da7a40-b868-485c-9d1f-fd85d3f1a252" (UID: "79da7a40-b868-485c-9d1f-fd85d3f1a252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:53.283143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.283126 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:53.284380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.284355 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca" pod "79da7a40-b868-485c-9d1f-fd85d3f1a252" (UID: "79da7a40-b868-485c-9d1f-fd85d3f1a252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:01:53.284464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.284420 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc" pod "79da7a40-b868-485c-9d1f-fd85d3f1a252" (UID: "79da7a40-b868-485c-9d1f-fd85d3f1a252"). InnerVolumeSpecName "kube-api-access-q2kcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:53.384344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.384289 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da7a40-b868-485c-9d1f-fd85d3f1a252-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:53.384344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.384310 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2kcc\" (UniqueName: \"kubernetes.io/projected/79da7a40-b868-485c-9d1f-fd85d3f1a252-kube-api-access-q2kcc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:01:53.849961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:53.849932 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:01:54.015919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:54.015879 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" Oct 29 22:01:54.016157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:54.015888 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9" event={"ID":"79da7a40-b868-485c-9d1f-fd85d3f1a252","Type":"ContainerDied","Data":"9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e"} Oct 29 22:01:54.016157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:54.016011 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b84458c03aa9fc955cf3f19ab01d39b27d799cbfa5bc65f69b3959eeaa28f7e" Oct 29 22:01:57.962726 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:01:57.962683 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:01:59.918513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:59.918482 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 22:01:59.958006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:59.957975 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 22:01:59.965907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:59.965887 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:01:59.966049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:01:59.965942 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:02:03.233969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.228937 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5"] Oct 29 22:02:03.233969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.230830 4543 cpu_manager.go:414] "containerMap: removing container" podUID="79da7a40-b868-485c-9d1f-fd85d3f1a252" containerName="revoke-keys" Oct 29 22:02:03.233969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.230852 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da7a40-b868-485c-9d1f-fd85d3f1a252" containerName="revoke-keys" Oct 29 22:02:03.233969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.231290 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da7a40-b868-485c-9d1f-fd85d3f1a252" containerName="revoke-keys" Oct 29 22:02:03.240131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.240107 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.256306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.256280 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5"] Oct 29 22:02:03.258359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.258332 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:02:03.258489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.258460 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wt9dr\"" Oct 29 22:02:03.258576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.258488 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:02:03.262760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.262739 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:02:03.346623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.346586 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.346991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.346964 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjd7\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-kube-api-access-2rjd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.347528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.347488 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.347634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.347554 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.347707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.347638 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.347791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.347733 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271c3c4-d946-4bce-afe7-1df9c7364966-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.347791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.347786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449192 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjd7\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-kube-api-access-2rjd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449317 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449371 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449447 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449526 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271c3c4-d946-4bce-afe7-1df9c7364966-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.449733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.449657 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.450538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.450507 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.454686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.453817 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.455548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.455522 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.455887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.455866 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.456498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.456460 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271c3c4-d946-4bce-afe7-1df9c7364966-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.475659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.475639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjd7\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-kube-api-access-2rjd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.475753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.475699 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271c3c4-d946-4bce-afe7-1df9c7364966-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v8ds5\" (UID: \"c271c3c4-d946-4bce-afe7-1df9c7364966\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.556549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.556500 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:03.887310 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:02:03.887282 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc271c3c4_d946_4bce_afe7_1df9c7364966.slice/crio-0e4f4c672a1289992ce88edf98af7b5bce7e34f9355b2676e8fc037137dcf644 WatchSource:0}: Error finding container 0e4f4c672a1289992ce88edf98af7b5bce7e34f9355b2676e8fc037137dcf644: Status 404 returned error can't find the container with id 0e4f4c672a1289992ce88edf98af7b5bce7e34f9355b2676e8fc037137dcf644 Oct 29 22:02:03.889450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.889384 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5"] Oct 29 22:02:03.889880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:03.889864 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:02:04.113175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.113140 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" event={"ID":"c271c3c4-d946-4bce-afe7-1df9c7364966","Type":"ContainerStarted","Data":"0e4f4c672a1289992ce88edf98af7b5bce7e34f9355b2676e8fc037137dcf644"} Oct 29 22:02:04.686515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.686483 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d"] Oct 29 22:02:04.692867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.692848 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.712889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.712865 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d"] Oct 29 22:02:04.719252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.719219 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:02:04.719404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.719269 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:02:04.719404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.719339 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:02:04.723484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.723460 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:02:04.867969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.867939 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.868178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.868136 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.868333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.868304 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54pw\" (UniqueName: \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.919778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.919734 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" podUID="156f8fb6-21dd-4c6b-b325-e1728d665493" containerName="payload-processing" containerID="cri-o://133e304deb1a83de1ae18222b7ff8a61b8f1c0303c95f8e98d7afc4384f737b1" gracePeriod=25 Oct 29 22:02:04.959246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.959209 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" podUID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" containerName="payload-pre-processing" containerID="cri-o://1272ea4b7ef129339c805d08256c2940b747b91c30c0f8400b85b1cb4f6e9448" gracePeriod=25 Oct 29 22:02:04.969525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.969499 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.969679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.969635 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54pw\" (UniqueName: \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.969849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.969831 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.970363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.970338 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.974463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.974440 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:04.996494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:04.996466 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54pw\" (UniqueName: \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:05.010323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.010304 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:05.132699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.132666 4543 generic.go:356] "Generic (PLEG): container finished" podID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" containerID="1272ea4b7ef129339c805d08256c2940b747b91c30c0f8400b85b1cb4f6e9448" exitCode=0 Oct 29 22:02:05.132979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.132957 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" event={"ID":"a541195a-7e55-4e2c-a0c4-daf3a92e11f3","Type":"ContainerDied","Data":"1272ea4b7ef129339c805d08256c2940b747b91c30c0f8400b85b1cb4f6e9448"} Oct 29 22:02:05.135400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.135248 4543 generic.go:356] "Generic (PLEG): container finished" podID="156f8fb6-21dd-4c6b-b325-e1728d665493" containerID="133e304deb1a83de1ae18222b7ff8a61b8f1c0303c95f8e98d7afc4384f737b1" exitCode=0 Oct 29 22:02:05.135400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.135345 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" event={"ID":"156f8fb6-21dd-4c6b-b325-e1728d665493","Type":"ContainerDied","Data":"133e304deb1a83de1ae18222b7ff8a61b8f1c0303c95f8e98d7afc4384f737b1"} Oct 29 22:02:05.356664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.356577 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d"] Oct 29 22:02:05.374951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.374922 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 22:02:05.382282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.382254 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\" (UniqueName: \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\") pod \"156f8fb6-21dd-4c6b-b325-e1728d665493\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " Oct 29 22:02:05.382394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.382311 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\") pod \"156f8fb6-21dd-4c6b-b325-e1728d665493\" (UID: \"156f8fb6-21dd-4c6b-b325-e1728d665493\") " Oct 29 22:02:05.383456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.383426 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume" pod "156f8fb6-21dd-4c6b-b325-e1728d665493" (UID: "156f8fb6-21dd-4c6b-b325-e1728d665493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:05.385791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.385763 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l" pod "156f8fb6-21dd-4c6b-b325-e1728d665493" (UID: "156f8fb6-21dd-4c6b-b325-e1728d665493"). InnerVolumeSpecName "kube-api-access-lgr9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:05.400110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.400057 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 22:02:05.484445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.484010 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\" (UniqueName: \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\") pod \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " Oct 29 22:02:05.484589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.484477 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\") pod \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\" (UID: \"a541195a-7e55-4e2c-a0c4-daf3a92e11f3\") " Oct 29 22:02:05.484941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.484866 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume" pod "a541195a-7e55-4e2c-a0c4-daf3a92e11f3" (UID: "a541195a-7e55-4e2c-a0c4-daf3a92e11f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:05.485627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.485603 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:05.485724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.485632 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgr9l\" (UniqueName: \"kubernetes.io/projected/156f8fb6-21dd-4c6b-b325-e1728d665493-kube-api-access-lgr9l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:05.485724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.485650 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f8fb6-21dd-4c6b-b325-e1728d665493-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:05.531459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.531425 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2" pod "a541195a-7e55-4e2c-a0c4-daf3a92e11f3" (UID: "a541195a-7e55-4e2c-a0c4-daf3a92e11f3"). InnerVolumeSpecName "kube-api-access-t26l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:05.587013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:05.586979 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26l2\" (UniqueName: \"kubernetes.io/projected/a541195a-7e55-4e2c-a0c4-daf3a92e11f3-kube-api-access-t26l2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:06.146240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.146151 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" event={"ID":"156f8fb6-21dd-4c6b-b325-e1728d665493","Type":"ContainerDied","Data":"63e0a32aadf6f26e7a59897fc839e9ce800f4054ce1291fff828de3e482f9fc9"} Oct 29 22:02:06.146240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.146176 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2" Oct 29 22:02:06.146720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.146269 4543 scope.go:122] "RemoveContainer" containerID="133e304deb1a83de1ae18222b7ff8a61b8f1c0303c95f8e98d7afc4384f737b1" Oct 29 22:02:06.147905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.147885 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" Oct 29 22:02:06.147905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.147894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7" event={"ID":"a541195a-7e55-4e2c-a0c4-daf3a92e11f3","Type":"ContainerDied","Data":"429ad95b9174c1a57da76ddfb66033652a857302007eb5773df8bc67e1cb5cf8"} Oct 29 22:02:06.150353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.150325 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" event={"ID":"76281216-2e36-4a55-8c4b-240b98a31357","Type":"ContainerStarted","Data":"eac2cce1d6fc654be4872c0df5c57340fc8170f2844b5504a8a0607ae34b90d8"} Oct 29 22:02:06.150483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.150375 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" event={"ID":"76281216-2e36-4a55-8c4b-240b98a31357","Type":"ContainerStarted","Data":"ab32f5560a6216b137ad6c35c6d8ec362c64d6886e6821196d50f67f716be53e"} Oct 29 22:02:06.161322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.161306 4543 scope.go:122] "RemoveContainer" containerID="1272ea4b7ef129339c805d08256c2940b747b91c30c0f8400b85b1cb4f6e9448" Oct 29 22:02:06.210620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.210592 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 22:02:06.220345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.220322 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7f9e0-b5f45975-xrzs7"] Oct 29 22:02:06.259424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.259399 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 22:02:06.264846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.264821 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7f9e0-5798f77f74-6v7n2"] Oct 29 22:02:06.308786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.308743 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" podStartSLOduration=2.308731301 podStartE2EDuration="2.308731301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:06.301620533 +0000 UTC m=+1188.043551206" watchObservedRunningTime="2026-10-29 22:02:06.308731301 +0000 UTC m=+1188.050661973" Oct 29 22:02:06.390191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.390160 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156f8fb6-21dd-4c6b-b325-e1728d665493" path="/var/lib/kubelet/pods/156f8fb6-21dd-4c6b-b325-e1728d665493/volumes" Oct 29 22:02:06.390792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.390763 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" path="/var/lib/kubelet/pods/a541195a-7e55-4e2c-a0c4-daf3a92e11f3/volumes" Oct 29 22:02:06.844767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.844727 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:02:06.844915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:06.844812 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:02:07.169649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.169542 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" event={"ID":"c271c3c4-d946-4bce-afe7-1df9c7364966","Type":"ContainerStarted","Data":"8d78fdc19176e4ab273dc4a9208793ffd1d51f062f43d89041fb0dcc3ba30278"} Oct 29 22:02:07.170218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.169875 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:07.172663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.172633 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-v8ds5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:02:07.172804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.172686 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" podUID="c271c3c4-d946-4bce-afe7-1df9c7364966" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:02:07.175538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.175509 4543 generic.go:356] "Generic (PLEG): container finished" podID="76281216-2e36-4a55-8c4b-240b98a31357" containerID="eac2cce1d6fc654be4872c0df5c57340fc8170f2844b5504a8a0607ae34b90d8" exitCode=0 Oct 29 22:02:07.175672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.175560 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" event={"ID":"76281216-2e36-4a55-8c4b-240b98a31357","Type":"ContainerDied","Data":"eac2cce1d6fc654be4872c0df5c57340fc8170f2844b5504a8a0607ae34b90d8"} Oct 29 22:02:07.236815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:07.236758 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" podStartSLOduration=1.283348236 podStartE2EDuration="4.236743373s" totalImagesPullingTime="2.953395137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:07.232009985 +0000 UTC m=+1188.973940659" watchObservedRunningTime="2026-10-29 22:02:07.236743373 +0000 UTC m=+1188.978674045" Oct 29 22:02:08.184527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.184382 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-v8ds5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:02:08.184527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.184435 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" podUID="c271c3c4-d946-4bce-afe7-1df9c7364966" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:02:08.352345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.352317 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:08.439298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.439238 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\") pod \"76281216-2e36-4a55-8c4b-240b98a31357\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " Oct 29 22:02:08.439436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.439421 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\") pod \"76281216-2e36-4a55-8c4b-240b98a31357\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " Oct 29 22:02:08.439509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.439493 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\" (UniqueName: \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\") pod \"76281216-2e36-4a55-8c4b-240b98a31357\" (UID: \"76281216-2e36-4a55-8c4b-240b98a31357\") " Oct 29 22:02:08.439845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.439817 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca" pod "76281216-2e36-4a55-8c4b-240b98a31357" (UID: "76281216-2e36-4a55-8c4b-240b98a31357"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:08.440239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.440218 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:08.441569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.441548 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca" pod "76281216-2e36-4a55-8c4b-240b98a31357" (UID: "76281216-2e36-4a55-8c4b-240b98a31357"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:08.442236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.442214 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw" pod "76281216-2e36-4a55-8c4b-240b98a31357" (UID: "76281216-2e36-4a55-8c4b-240b98a31357"). InnerVolumeSpecName "kube-api-access-c54pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:08.542087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.542051 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76281216-2e36-4a55-8c4b-240b98a31357-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:08.542196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:08.542113 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54pw\" (UniqueName: \"kubernetes.io/projected/76281216-2e36-4a55-8c4b-240b98a31357-kube-api-access-c54pw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:09.197413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:09.197378 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" Oct 29 22:02:09.197799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:09.197408 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d" event={"ID":"76281216-2e36-4a55-8c4b-240b98a31357","Type":"ContainerDied","Data":"ab32f5560a6216b137ad6c35c6d8ec362c64d6886e6821196d50f67f716be53e"} Oct 29 22:02:09.197799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:09.197487 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab32f5560a6216b137ad6c35c6d8ec362c64d6886e6821196d50f67f716be53e" Oct 29 22:02:11.184849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:11.184770 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v8ds5" Oct 29 22:02:15.240300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:15.240263 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 22:02:18.972906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:18.972876 4543 scope.go:122] "RemoveContainer" containerID="acb2519768f4a8cbd95d9eaf2ca7cf1d6c780654d921d2e93a41a8ddb3c42983" Oct 29 22:02:18.983721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:18.983697 4543 scope.go:122] "RemoveContainer" containerID="8e69bc6a48e0e216decc518c00769b7d0f97053c82ef4e2a93f3231bc894ce79" Oct 29 22:02:18.993849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:18.993819 4543 scope.go:122] "RemoveContainer" containerID="cbfdd74cf75f382cb5114046f902376c6fae21ae7198576a3ae0b0f3425529e1" Oct 29 22:02:19.003629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:19.003607 4543 scope.go:122] "RemoveContainer" containerID="29a98069ff447468ede39a6c2f14baff7110d19a5409af8002a42ad040eacdf2" Oct 29 22:02:20.240869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:20.240827 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" containerID="cri-o://3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7" gracePeriod=25 Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.391233 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4"] Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392442 4543 cpu_manager.go:414] "containerMap: removing container" podUID="156f8fb6-21dd-4c6b-b325-e1728d665493" containerName="payload-processing" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392464 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f8fb6-21dd-4c6b-b325-e1728d665493" containerName="payload-processing" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392518 4543 cpu_manager.go:414] "containerMap: removing container" podUID="76281216-2e36-4a55-8c4b-240b98a31357" containerName="revoke-keys" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392528 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76281216-2e36-4a55-8c4b-240b98a31357" containerName="revoke-keys" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392551 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" containerName="payload-pre-processing" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392559 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" containerName="payload-pre-processing" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392770 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="76281216-2e36-4a55-8c4b-240b98a31357" containerName="revoke-keys" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392791 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a541195a-7e55-4e2c-a0c4-daf3a92e11f3" containerName="payload-pre-processing" Oct 29 22:02:21.395119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.392813 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f8fb6-21dd-4c6b-b325-e1728d665493" containerName="payload-processing" Oct 29 22:02:21.422776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.421352 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.452102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.447038 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:02:21.492325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.484580 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:02:21.492325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.484705 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:02:21.492325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.484811 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:02:21.525187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.525155 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4"] Oct 29 22:02:21.734102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.729475 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxmw\" (UniqueName: \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.734102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.729738 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.734102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.729943 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.838100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.835268 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.838100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.835445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxmw\" (UniqueName: \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.838100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.835630 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:21.861187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.859607 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:22.004152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:21.995910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxmw\" (UniqueName: \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:22.004152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.001501 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f00ced5-lkdn4\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:22.102125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.102067 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:22.794137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.783373 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:22.794137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.783782 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:02:22.794137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.784126 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:02:22.794137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.785913 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:02:22.836452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.836411 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:02:22.837693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:22.837665 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.357488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.357445 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" podUID="543853b5-9761-4852-8801-1519786ced31" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.392145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.391568 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.615362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.615262 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.673157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.669963 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4"] Oct 29 22:02:23.727119 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:02:23.727063 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429e313e_6058_4893_913a_d7483f70a346.slice/crio-9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41 WatchSource:0}: Error finding container 9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41: Status 404 returned error can't find the container with id 9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41 Oct 29 22:02:23.757109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.751308 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" podUID="543853b5-9761-4852-8801-1519786ced31" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.784727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.784671 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.786587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.786561 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:02:23.807098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.787974 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.820866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.796307 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.867818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.821127 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:02:23.867818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.821195 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:02:23.867818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.864887 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:02:23.867818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.865605 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:02:23.885781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.870588 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:02:23.885781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.870684 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:02:23.920240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.920201 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:02:23.921142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:23.921120 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:02:24.084179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.068708 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:02:24.101842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.101815 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:02:24.105249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.105137 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:02:24.125273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.125215 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:02:24.187350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.187323 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:02:24.187633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.187613 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:02:24.274691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.274545 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:02:24.287280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.287253 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:02:24.315630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.315579 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:02:24.679763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:24.679731 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" event={"ID":"429e313e-6058-4893-913a-d7483f70a346","Type":"ContainerStarted","Data":"9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41"} Oct 29 22:02:25.690187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:25.690102 4543 generic.go:356] "Generic (PLEG): container finished" podID="429e313e-6058-4893-913a-d7483f70a346" containerID="c3ca949f0a1549dcf224f02f2fe550c19fae2e188accf63a7b0343ef12012640" exitCode=0 Oct 29 22:02:25.690591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:25.690249 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" event={"ID":"429e313e-6058-4893-913a-d7483f70a346","Type":"ContainerDied","Data":"c3ca949f0a1549dcf224f02f2fe550c19fae2e188accf63a7b0343ef12012640"} Oct 29 22:02:27.100002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.099971 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:27.194672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.193260 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\" (UniqueName: \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\") pod \"429e313e-6058-4893-913a-d7483f70a346\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " Oct 29 22:02:27.194672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.193630 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\") pod \"429e313e-6058-4893-913a-d7483f70a346\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " Oct 29 22:02:27.194672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.193671 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\") pod \"429e313e-6058-4893-913a-d7483f70a346\" (UID: \"429e313e-6058-4893-913a-d7483f70a346\") " Oct 29 22:02:27.195556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.195504 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca" pod "429e313e-6058-4893-913a-d7483f70a346" (UID: "429e313e-6058-4893-913a-d7483f70a346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:27.200520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.200392 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw" pod "429e313e-6058-4893-913a-d7483f70a346" (UID: "429e313e-6058-4893-913a-d7483f70a346"). InnerVolumeSpecName "kube-api-access-gtxmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:27.203007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.202982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca" pod "429e313e-6058-4893-913a-d7483f70a346" (UID: "429e313e-6058-4893-913a-d7483f70a346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:27.275735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.275703 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf"] Oct 29 22:02:27.280866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.280838 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4082f-fgspf"] Oct 29 22:02:27.295726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.295701 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429e313e-6058-4893-913a-d7483f70a346-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.295726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.295728 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429e313e-6058-4893-913a-d7483f70a346-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.295869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.295740 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxmw\" (UniqueName: \"kubernetes.io/projected/429e313e-6058-4893-913a-d7483f70a346-kube-api-access-gtxmw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:27.764869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.764842 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" Oct 29 22:02:27.764869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.764858 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4" event={"ID":"429e313e-6058-4893-913a-d7483f70a346","Type":"ContainerDied","Data":"9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41"} Oct 29 22:02:27.765125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:27.764906 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee69d72bc6b4a84f1277ee2e8a68ca83059adfbbabe441f159b9b34b1971d41" Oct 29 22:02:27.961968 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:02:27.961930 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:02:28.392015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:28.391976 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a8a2b3-d7c7-48d5-803a-e149a9019f6c" path="/var/lib/kubelet/pods/08a8a2b3-d7c7-48d5-803a-e149a9019f6c/volumes" Oct 29 22:02:28.625775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:28.625741 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:02:28.670619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:28.670339 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:02:29.276790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.276680 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" podUID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" containerName="payload-processing" containerID="cri-o://1a1ae2cbe20b4f8f37a9e6676728b170c55de66195b4e13ad7896dfcd5420575" gracePeriod=25 Oct 29 22:02:29.287793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.287758 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" podUID="be46f751-6e61-4767-934d-9df40fc58f5e" containerName="payload-pre-processing" containerID="cri-o://8b1413fcfec30f53f5e41a38f36f52a0daece7a98ccb5a73867fcd8bba8f4cab" gracePeriod=25 Oct 29 22:02:29.807648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.807524 4543 generic.go:356] "Generic (PLEG): container finished" podID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" containerID="1a1ae2cbe20b4f8f37a9e6676728b170c55de66195b4e13ad7896dfcd5420575" exitCode=0 Oct 29 22:02:29.808232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.807694 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" event={"ID":"01f6becd-2509-4dc8-828e-f0a1b00dbc92","Type":"ContainerDied","Data":"1a1ae2cbe20b4f8f37a9e6676728b170c55de66195b4e13ad7896dfcd5420575"} Oct 29 22:02:29.808232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.807757 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" event={"ID":"01f6becd-2509-4dc8-828e-f0a1b00dbc92","Type":"ContainerDied","Data":"d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a"} Oct 29 22:02:29.808232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.807785 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e52c1f845793b0573de4c10cb9d28716a94d926de1e31dc421fe65fed9ee7a" Oct 29 22:02:29.811160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.811135 4543 generic.go:356] "Generic (PLEG): container finished" podID="be46f751-6e61-4767-934d-9df40fc58f5e" containerID="8b1413fcfec30f53f5e41a38f36f52a0daece7a98ccb5a73867fcd8bba8f4cab" exitCode=0 Oct 29 22:02:29.811433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.811413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" event={"ID":"be46f751-6e61-4767-934d-9df40fc58f5e","Type":"ContainerDied","Data":"8b1413fcfec30f53f5e41a38f36f52a0daece7a98ccb5a73867fcd8bba8f4cab"} Oct 29 22:02:29.812156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.812048 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:02:29.854537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.854510 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:02:29.895388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.893197 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\") pod \"be46f751-6e61-4767-934d-9df40fc58f5e\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " Oct 29 22:02:29.895388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.893313 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\" (UniqueName: \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\") pod \"be46f751-6e61-4767-934d-9df40fc58f5e\" (UID: \"be46f751-6e61-4767-934d-9df40fc58f5e\") " Oct 29 22:02:29.895388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.893421 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\" (UniqueName: \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\") pod \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " Oct 29 22:02:29.895388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.893448 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\") pod \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\" (UID: \"01f6becd-2509-4dc8-828e-f0a1b00dbc92\") " Oct 29 22:02:29.895388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.894984 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume" pod "01f6becd-2509-4dc8-828e-f0a1b00dbc92" (UID: "01f6becd-2509-4dc8-828e-f0a1b00dbc92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:29.895867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.895841 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume" pod "be46f751-6e61-4767-934d-9df40fc58f5e" (UID: "be46f751-6e61-4767-934d-9df40fc58f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:29.898399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.898375 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq" pod "be46f751-6e61-4767-934d-9df40fc58f5e" (UID: "be46f751-6e61-4767-934d-9df40fc58f5e"). InnerVolumeSpecName "kube-api-access-8jpbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:29.899491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.899460 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk" pod "01f6becd-2509-4dc8-828e-f0a1b00dbc92" (UID: "01f6becd-2509-4dc8-828e-f0a1b00dbc92"). InnerVolumeSpecName "kube-api-access-kkvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:29.995948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.995914 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be46f751-6e61-4767-934d-9df40fc58f5e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:29.995948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.995949 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpbq\" (UniqueName: \"kubernetes.io/projected/be46f751-6e61-4767-934d-9df40fc58f5e-kube-api-access-8jpbq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:29.996176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.995966 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvwk\" (UniqueName: \"kubernetes.io/projected/01f6becd-2509-4dc8-828e-f0a1b00dbc92-kube-api-access-kkvwk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:29.996176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:29.995980 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f6becd-2509-4dc8-828e-f0a1b00dbc92-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:30.823572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.823529 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" Oct 29 22:02:30.824132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.823569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t" event={"ID":"be46f751-6e61-4767-934d-9df40fc58f5e","Type":"ContainerDied","Data":"357ea9c3a0b1d7006e36dfcaf8aad05c799c0427ef7f2b29779252a2a05b8b24"} Oct 29 22:02:30.824132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.823648 4543 scope.go:122] "RemoveContainer" containerID="8b1413fcfec30f53f5e41a38f36f52a0daece7a98ccb5a73867fcd8bba8f4cab" Oct 29 22:02:30.824132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.823691 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774" Oct 29 22:02:30.902193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.902165 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:02:30.910113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.910091 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76c786-6d7bd96698-6h774"] Oct 29 22:02:30.951628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.951601 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:02:30.958885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:30.958855 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76c786-5bb95fb55f-pvw6t"] Oct 29 22:02:31.402724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.401540 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": read tcp 10.133.0.2:41434->10.133.0.139:15021: read: connection reset by peer" start-of-body= Oct 29 22:02:31.402724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.401599 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": read tcp 10.133.0.2:41434->10.133.0.139:15021: read: connection reset by peer" Oct 29 22:02:31.834941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.834909 4543 generic.go:356] "Generic (PLEG): container finished" podID="81029b9d-4392-4638-ad49-600ac5dd1996" containerID="2618bc8622c7a9a403dcdfe914d49ef2da301be712628f8d8e3d3b3445ce1bec" exitCode=0 Oct 29 22:02:31.835337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.834983 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerDied","Data":"2618bc8622c7a9a403dcdfe914d49ef2da301be712628f8d8e3d3b3445ce1bec"} Oct 29 22:02:31.835385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.835347 4543 scope.go:122] "RemoveContainer" containerID="2618bc8622c7a9a403dcdfe914d49ef2da301be712628f8d8e3d3b3445ce1bec" Oct 29 22:02:31.836861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.836830 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:02:31.837022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.836901 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:02:31.837022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:31.836936 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:02:32.387203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:32.387169 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" path="/var/lib/kubelet/pods/01f6becd-2509-4dc8-828e-f0a1b00dbc92/volumes" Oct 29 22:02:32.387651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:32.387637 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be46f751-6e61-4767-934d-9df40fc58f5e" path="/var/lib/kubelet/pods/be46f751-6e61-4767-934d-9df40fc58f5e/volumes" Oct 29 22:02:32.858857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:32.858798 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerStarted","Data":"b2c494824d5e32eeebb116ed0da3bcff505c96cdc635a4da72fa61ed2ae493db"} Oct 29 22:02:33.159060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.158976 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:02:33.159337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.159207 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:33.159457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.159370 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:33.627028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.626965 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" containerName="payload-processing" containerID="cri-o://fce248f8e6ad81c084fc643e3648dc8b33a79f8f93ccf112029bd68739edca60" gracePeriod=25 Oct 29 22:02:33.674640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.673233 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" podUID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" containerName="payload-pre-processing" containerID="cri-o://3f18b1fbf37c9c1ab0169d9102170a80b0044d5d1fb66b8d9e41b46bf5e3a721" gracePeriod=25 Oct 29 22:02:33.855536 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:02:33.855508 4543 logging.go:55] [core] [Channel #2477 SubChannel #2478]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.144:50052", ServerName: "10.133.0.144:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.144:50052: connect: connection refused" Oct 29 22:02:33.882779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.882635 4543 generic.go:356] "Generic (PLEG): container finished" podID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" containerID="3f18b1fbf37c9c1ab0169d9102170a80b0044d5d1fb66b8d9e41b46bf5e3a721" exitCode=0 Oct 29 22:02:33.882779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.882704 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" event={"ID":"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76","Type":"ContainerDied","Data":"3f18b1fbf37c9c1ab0169d9102170a80b0044d5d1fb66b8d9e41b46bf5e3a721"} Oct 29 22:02:33.885652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.885623 4543 generic.go:356] "Generic (PLEG): container finished" podID="79ea458d-3041-41c3-8159-c9838eb1173e" containerID="fce248f8e6ad81c084fc643e3648dc8b33a79f8f93ccf112029bd68739edca60" exitCode=0 Oct 29 22:02:33.886326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:33.886303 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" event={"ID":"79ea458d-3041-41c3-8159-c9838eb1173e","Type":"ContainerDied","Data":"fce248f8e6ad81c084fc643e3648dc8b33a79f8f93ccf112029bd68739edca60"} Oct 29 22:02:34.077141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.077111 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:02:34.127153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.125690 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\") pod \"79ea458d-3041-41c3-8159-c9838eb1173e\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " Oct 29 22:02:34.127153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.125836 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\" (UniqueName: \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\") pod \"79ea458d-3041-41c3-8159-c9838eb1173e\" (UID: \"79ea458d-3041-41c3-8159-c9838eb1173e\") " Oct 29 22:02:34.148554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.127853 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume" pod "79ea458d-3041-41c3-8159-c9838eb1173e" (UID: "79ea458d-3041-41c3-8159-c9838eb1173e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:34.160267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.159346 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb" pod "79ea458d-3041-41c3-8159-c9838eb1173e" (UID: "79ea458d-3041-41c3-8159-c9838eb1173e"). InnerVolumeSpecName "kube-api-access-6tjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:34.160267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.159691 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:34.160267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.159747 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:34.169765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.169741 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:02:34.227631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.227603 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\" (UniqueName: \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\") pod \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " Oct 29 22:02:34.227782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.227655 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\") pod \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\" (UID: \"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76\") " Oct 29 22:02:34.228208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.228144 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume" pod "bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" (UID: "bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:34.231111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.228480 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tjhb\" (UniqueName: \"kubernetes.io/projected/79ea458d-3041-41c3-8159-c9838eb1173e-kube-api-access-6tjhb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.231111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.228508 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.231111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.228535 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ea458d-3041-41c3-8159-c9838eb1173e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.251466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.251431 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8" pod "bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" (UID: "bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76"). InnerVolumeSpecName "kube-api-access-lj8z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:34.315356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.315320 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:02:34.354585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.354548 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8z8\" (UniqueName: \"kubernetes.io/projected/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76-kube-api-access-lj8z8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:34.855462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.855421 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.144:50052\" within 1s: context deadline exceeded" Oct 29 22:02:34.895024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.894993 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" Oct 29 22:02:34.895495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.895032 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c" event={"ID":"bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76","Type":"ContainerDied","Data":"a9b8336a5d03f6c6ee57caf0a440d4bc5ab56cc9a4965b996a55fb76a1e50cd7"} Oct 29 22:02:34.895495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.895114 4543 scope.go:122] "RemoveContainer" containerID="3f18b1fbf37c9c1ab0169d9102170a80b0044d5d1fb66b8d9e41b46bf5e3a721" Oct 29 22:02:34.896969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.896949 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" Oct 29 22:02:34.897207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.896987 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z" event={"ID":"79ea458d-3041-41c3-8159-c9838eb1173e","Type":"ContainerDied","Data":"4e2ab2ad1b111f6b5d23300dddd2c90f387d78a7772f6ca6fc04cc0350f2ec3c"} Oct 29 22:02:34.906637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.906621 4543 scope.go:122] "RemoveContainer" containerID="fce248f8e6ad81c084fc643e3648dc8b33a79f8f93ccf112029bd68739edca60" Oct 29 22:02:34.977821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.977796 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:02:34.994780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:34.994760 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-fb47c9775-x545c"] Oct 29 22:02:35.022004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.021978 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:02:35.088110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.087385 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fd295-885687478-xxv4z"] Oct 29 22:02:35.109225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.097937 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz"] Oct 29 22:02:35.145101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.144957 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba698a2c-7zcgz"] Oct 29 22:02:35.168101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.166506 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:35.168101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.166569 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:35.394154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.392113 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9"] Oct 29 22:02:35.398114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:35.397913 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7f9e0-6ttk9"] Oct 29 22:02:36.162435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.161493 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:36.162435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.161551 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:36.387341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.387311 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0179a221-f690-4f0b-b3cc-81d1cfa03b4f" path="/var/lib/kubelet/pods/0179a221-f690-4f0b-b3cc-81d1cfa03b4f/volumes" Oct 29 22:02:36.387854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.387839 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da7a40-b868-485c-9d1f-fd85d3f1a252" path="/var/lib/kubelet/pods/79da7a40-b868-485c-9d1f-fd85d3f1a252/volumes" Oct 29 22:02:36.388291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.388277 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" path="/var/lib/kubelet/pods/79ea458d-3041-41c3-8159-c9838eb1173e/volumes" Oct 29 22:02:36.389009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:36.388993 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" path="/var/lib/kubelet/pods/bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76/volumes" Oct 29 22:02:37.159433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.159393 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:37.159631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.159450 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:37.197212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.197178 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g"] Oct 29 22:02:37.198259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198241 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" containerName="payload-pre-processing" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198262 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" containerName="payload-pre-processing" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198277 4543 cpu_manager.go:414] "containerMap: removing container" podUID="429e313e-6058-4893-913a-d7483f70a346" containerName="revoke-keys" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198284 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429e313e-6058-4893-913a-d7483f70a346" containerName="revoke-keys" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198301 4543 cpu_manager.go:414] "containerMap: removing container" podUID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" containerName="payload-processing" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198308 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" containerName="payload-processing" Oct 29 22:02:37.198332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198328 4543 cpu_manager.go:414] "containerMap: removing container" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" containerName="payload-processing" Oct 29 22:02:37.198515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198336 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" containerName="payload-processing" Oct 29 22:02:37.198515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198349 4543 cpu_manager.go:414] "containerMap: removing container" podUID="be46f751-6e61-4767-934d-9df40fc58f5e" containerName="payload-pre-processing" Oct 29 22:02:37.198515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198357 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be46f751-6e61-4767-934d-9df40fc58f5e" containerName="payload-pre-processing" Oct 29 22:02:37.198615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198531 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf91cf25-f4d7-4fc2-bbf9-1f857cc33b76" containerName="payload-pre-processing" Oct 29 22:02:37.198615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198546 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="429e313e-6058-4893-913a-d7483f70a346" containerName="revoke-keys" Oct 29 22:02:37.198615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198555 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f6becd-2509-4dc8-828e-f0a1b00dbc92" containerName="payload-processing" Oct 29 22:02:37.198615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198564 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ea458d-3041-41c3-8159-c9838eb1173e" containerName="payload-processing" Oct 29 22:02:37.198615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.198573 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="be46f751-6e61-4767-934d-9df40fc58f5e" containerName="payload-pre-processing" Oct 29 22:02:37.203856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.203837 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.222556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.222523 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:02:37.222712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.222695 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:02:37.223151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.223133 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g"] Oct 29 22:02:37.274404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.274376 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:02:37.275341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.275317 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:02:37.384110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.384061 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.384110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.384111 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.384312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.384268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg9hb\" (UniqueName: \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.485289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.485260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg9hb\" (UniqueName: \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.485498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.485482 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.485563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.485509 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.486177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.486150 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.488679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.488658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.564098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.551827 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg9hb\" (UniqueName: \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\") pod \"maas-api-revoke-keys-e2e-api-b-579f09-zv42g\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:37.819784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:37.819710 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:38.159003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:38.158839 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 22:02:38.159003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:38.158893 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 22:02:38.210410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:38.210383 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g"] Oct 29 22:02:38.210410 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:02:38.210395 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cfe408_d146_4ca9_a29d_b23850eb6cab.slice/crio-d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c WatchSource:0}: Error finding container d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c: Status 404 returned error can't find the container with id d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c Oct 29 22:02:38.975210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:38.975174 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" event={"ID":"55cfe408-d146-4ca9-a29d-b23850eb6cab","Type":"ContainerStarted","Data":"f3b64c4e25e8e4e04cd18a375091487d60b150be58d9c4e67f4e2d1acaf7a68e"} Oct 29 22:02:38.975376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:38.975229 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" event={"ID":"55cfe408-d146-4ca9-a29d-b23850eb6cab","Type":"ContainerStarted","Data":"d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c"} Oct 29 22:02:39.022764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.022705 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" podStartSLOduration=2.022689334 podStartE2EDuration="2.022689334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:39.021427438 +0000 UTC m=+1220.763358111" watchObservedRunningTime="2026-10-29 22:02:39.022689334 +0000 UTC m=+1220.764620008" Oct 29 22:02:39.047789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.046941 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c"] Oct 29 22:02:39.089792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.089749 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4028a4e-4jw8c"] Oct 29 22:02:39.164097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.164048 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:02:39.164609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.164529 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:02:39.165540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.165446 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:02:39.986714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.986677 4543 generic.go:356] "Generic (PLEG): container finished" podID="55cfe408-d146-4ca9-a29d-b23850eb6cab" containerID="f3b64c4e25e8e4e04cd18a375091487d60b150be58d9c4e67f4e2d1acaf7a68e" exitCode=0 Oct 29 22:02:39.987266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:39.987043 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" event={"ID":"55cfe408-d146-4ca9-a29d-b23850eb6cab","Type":"ContainerDied","Data":"f3b64c4e25e8e4e04cd18a375091487d60b150be58d9c4e67f4e2d1acaf7a68e"} Oct 29 22:02:40.387767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.387731 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c727f81-937b-46c6-84d3-71d0efde3419" path="/var/lib/kubelet/pods/9c727f81-937b-46c6-84d3-71d0efde3419/volumes" Oct 29 22:02:40.521641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.521616 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 22:02:40.551375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.551345 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\") pod \"214d1999-43cc-48c3-aee6-b9210eaea845\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " Oct 29 22:02:40.551504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.551406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\" (UniqueName: \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\") pod \"214d1999-43cc-48c3-aee6-b9210eaea845\" (UID: \"214d1999-43cc-48c3-aee6-b9210eaea845\") " Oct 29 22:02:40.551714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.551689 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume" pod "214d1999-43cc-48c3-aee6-b9210eaea845" (UID: "214d1999-43cc-48c3-aee6-b9210eaea845"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:40.552043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.552020 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214d1999-43cc-48c3-aee6-b9210eaea845-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:40.553684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.553661 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks" pod "214d1999-43cc-48c3-aee6-b9210eaea845" (UID: "214d1999-43cc-48c3-aee6-b9210eaea845"). InnerVolumeSpecName "kube-api-access-5g4ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:40.654428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.654396 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4ks\" (UniqueName: \"kubernetes.io/projected/214d1999-43cc-48c3-aee6-b9210eaea845-kube-api-access-5g4ks\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:40.996382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.996353 4543 generic.go:356] "Generic (PLEG): container finished" podID="214d1999-43cc-48c3-aee6-b9210eaea845" containerID="3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7" exitCode=0 Oct 29 22:02:40.996769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.996398 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" Oct 29 22:02:40.996769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.996436 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" event={"ID":"214d1999-43cc-48c3-aee6-b9210eaea845","Type":"ContainerDied","Data":"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7"} Oct 29 22:02:40.996769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.996489 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q" event={"ID":"214d1999-43cc-48c3-aee6-b9210eaea845","Type":"ContainerDied","Data":"e692a9efb49e82e962f478451fe92862a1dbc127e207990084196871b3970787"} Oct 29 22:02:40.996769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:40.996518 4543 scope.go:122] "RemoveContainer" containerID="3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7" Oct 29 22:02:41.009653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.009633 4543 scope.go:122] "RemoveContainer" containerID="3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7" Oct 29 22:02:41.009914 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:02:41.009894 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7\": container with ID starting with 3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7 not found: ID does not exist" containerID="3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7" Oct 29 22:02:41.009985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.009924 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7"} err="failed to get container status \"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7\": rpc error: code = NotFound desc = could not find container \"3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7\": container with ID starting with 3f0f213c8e910a0d59d3546cfee6888a13fbb83bb482f488f205c28d81a586e7 not found: ID does not exist" Oct 29 22:02:41.079428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.079395 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 22:02:41.090286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.090256 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bd85q"] Oct 29 22:02:41.179284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.179260 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:41.267847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.267772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\" (UniqueName: \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\") pod \"55cfe408-d146-4ca9-a29d-b23850eb6cab\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " Oct 29 22:02:41.268046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.268024 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\") pod \"55cfe408-d146-4ca9-a29d-b23850eb6cab\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " Oct 29 22:02:41.268235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.268211 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\") pod \"55cfe408-d146-4ca9-a29d-b23850eb6cab\" (UID: \"55cfe408-d146-4ca9-a29d-b23850eb6cab\") " Oct 29 22:02:41.269009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.268820 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca" pod "55cfe408-d146-4ca9-a29d-b23850eb6cab" (UID: "55cfe408-d146-4ca9-a29d-b23850eb6cab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:41.269009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.268979 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.270457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.270431 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb" pod "55cfe408-d146-4ca9-a29d-b23850eb6cab" (UID: "55cfe408-d146-4ca9-a29d-b23850eb6cab"). InnerVolumeSpecName "kube-api-access-vg9hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:41.270751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.270735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca" pod "55cfe408-d146-4ca9-a29d-b23850eb6cab" (UID: "55cfe408-d146-4ca9-a29d-b23850eb6cab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:02:41.370623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.370588 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55cfe408-d146-4ca9-a29d-b23850eb6cab-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:41.370623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:41.370622 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg9hb\" (UniqueName: \"kubernetes.io/projected/55cfe408-d146-4ca9-a29d-b23850eb6cab-kube-api-access-vg9hb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:42.008929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:42.008897 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" Oct 29 22:02:42.008929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:42.008914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g" event={"ID":"55cfe408-d146-4ca9-a29d-b23850eb6cab","Type":"ContainerDied","Data":"d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c"} Oct 29 22:02:42.009428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:42.008978 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1361660905a82bd06082dded76a201905046c5921edc3e09f9a2812af00719c" Oct 29 22:02:42.387245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:42.387155 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" path="/var/lib/kubelet/pods/214d1999-43cc-48c3-aee6-b9210eaea845/volumes" Oct 29 22:02:43.693923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:43.693829 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 22:02:43.694681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:43.694654 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" containerID="cri-o://738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40" gracePeriod=30 Oct 29 22:02:46.037671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:46.037636 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg"] Oct 29 22:02:46.052646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:46.052619 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096719ca-cvdpg"] Oct 29 22:02:46.387262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:46.387176 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2964a666-ab61-41e7-b152-8ec68408bd5d" path="/var/lib/kubelet/pods/2964a666-ab61-41e7-b152-8ec68408bd5d/volumes" Oct 29 22:02:47.093138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:47.093053 4543 patch_prober.go:29] interesting pod/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:02:47.093712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:47.093176 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:02:47.969703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:47.969281 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:02:47.973206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:47.973178 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d"] Oct 29 22:02:47.978588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:47.978553 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-2xd8d"] Oct 29 22:02:48.011778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:48.011751 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:02:48.389781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:48.389696 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76281216-2e36-4a55-8c4b-240b98a31357" path="/var/lib/kubelet/pods/76281216-2e36-4a55-8c4b-240b98a31357/volumes" Oct 29 22:02:49.089030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.089003 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:02:49.094727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.094690 4543 generic.go:356] "Generic (PLEG): container finished" podID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerID="738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40" exitCode=0 Oct 29 22:02:49.095230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.095201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerDied","Data":"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40"} Oct 29 22:02:49.095615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.095401 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" Oct 29 22:02:49.095615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.095416 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4" event={"ID":"cd17290b-d871-4cd2-9843-198f3d1dfe68","Type":"ContainerDied","Data":"cda6553985856dcfb59c8d599942f93954548a8d2b776c79ebf7dcab3b51fa3b"} Oct 29 22:02:49.095615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.095462 4543 scope.go:122] "RemoveContainer" containerID="738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40" Oct 29 22:02:49.113953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.113928 4543 scope.go:122] "RemoveContainer" containerID="b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09" Oct 29 22:02:49.125703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.125681 4543 scope.go:122] "RemoveContainer" containerID="738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40" Oct 29 22:02:49.126906 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:02:49.125974 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40\": container with ID starting with 738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40 not found: ID does not exist" containerID="738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40" Oct 29 22:02:49.126906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.126012 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40"} err="failed to get container status \"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40\": rpc error: code = NotFound desc = could not find container \"738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40\": container with ID starting with 738f00e2bb4693628e4ee62583aa07e0d98c0e411a0b4cbda37e6d025d8b7e40 not found: ID does not exist" Oct 29 22:02:49.126906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.126041 4543 scope.go:122] "RemoveContainer" containerID="b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09" Oct 29 22:02:49.126906 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:02:49.126342 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09\": container with ID starting with b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09 not found: ID does not exist" containerID="b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09" Oct 29 22:02:49.126906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.126371 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09"} err="failed to get container status \"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09\": rpc error: code = NotFound desc = could not find container \"b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09\": container with ID starting with b8ce1e474e94f7e1a4c799f6891ee6f4cb7dd2f08c1983b59999223e8c2d3e09 not found: ID does not exist" Oct 29 22:02:49.161449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161425 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.161603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161478 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.161685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161623 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.161752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161721 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:49.161849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161798 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.161909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.161967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161948 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.162031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.161993 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.162106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162031 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.162106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162069 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\") pod \"cd17290b-d871-4cd2-9843-198f3d1dfe68\" (UID: \"cd17290b-d871-4cd2-9843-198f3d1dfe68\") " Oct 29 22:02:49.162222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162150 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:49.162560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162409 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:49.162560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:49.162800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.162623 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:49.163255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.163215 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.163255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.163242 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd17290b-d871-4cd2-9843-198f3d1dfe68-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.163502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.163257 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.163502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.163274 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.163502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.163287 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.165017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.164982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:49.165364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.165341 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:02:49.165498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.165474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:02:49.165637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.165621 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv" pod "cd17290b-d871-4cd2-9843-198f3d1dfe68" (UID: "cd17290b-d871-4cd2-9843-198f3d1dfe68"). InnerVolumeSpecName "kube-api-access-5w9zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:49.264413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.264386 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.264413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.264412 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9zv\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-kube-api-access-5w9zv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.264555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.264421 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.264555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.264430 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd17290b-d871-4cd2-9843-198f3d1dfe68-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:49.455134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.455093 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 22:02:49.468088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:49.468055 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d7f9e0-openshift-default-76d4d798d9-46hb4"] Oct 29 22:02:50.386946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:50.386902 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" path="/var/lib/kubelet/pods/cd17290b-d871-4cd2-9843-198f3d1dfe68/volumes" Oct 29 22:02:52.970419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:52.970364 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" podUID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" containerName="payload-processing" containerID="cri-o://6af8eedd65e0764d9bcd17ba3d032c492e36ee072d99975c1541b7735507b2aa" gracePeriod=25 Oct 29 22:02:53.012682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.012643 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" podUID="cd752ac0-8178-4e74-8e58-a8842d6909d5" containerName="payload-pre-processing" containerID="cri-o://002e23bfa8761694cb858b785f8d2107e83dacc162900dc7116cac3c71d57b63" gracePeriod=25 Oct 29 22:02:53.163469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.163436 4543 generic.go:356] "Generic (PLEG): container finished" podID="cd752ac0-8178-4e74-8e58-a8842d6909d5" containerID="002e23bfa8761694cb858b785f8d2107e83dacc162900dc7116cac3c71d57b63" exitCode=0 Oct 29 22:02:53.163622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.163566 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" event={"ID":"cd752ac0-8178-4e74-8e58-a8842d6909d5","Type":"ContainerDied","Data":"002e23bfa8761694cb858b785f8d2107e83dacc162900dc7116cac3c71d57b63"} Oct 29 22:02:53.165793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.165638 4543 generic.go:356] "Generic (PLEG): container finished" podID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" containerID="6af8eedd65e0764d9bcd17ba3d032c492e36ee072d99975c1541b7735507b2aa" exitCode=0 Oct 29 22:02:53.165793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.165731 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" event={"ID":"816c43f4-5339-4fc1-92d4-9f15f53d5ac2","Type":"ContainerDied","Data":"6af8eedd65e0764d9bcd17ba3d032c492e36ee072d99975c1541b7735507b2aa"} Oct 29 22:02:53.313541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.313511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:02:53.344411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.344391 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:02:53.458575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.458543 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\") pod \"cd752ac0-8178-4e74-8e58-a8842d6909d5\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " Oct 29 22:02:53.458819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.458584 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\") pod \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " Oct 29 22:02:53.458900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.458834 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\" (UniqueName: \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\") pod \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\" (UID: \"816c43f4-5339-4fc1-92d4-9f15f53d5ac2\") " Oct 29 22:02:53.458972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.458949 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume" pod "cd752ac0-8178-4e74-8e58-a8842d6909d5" (UID: "cd752ac0-8178-4e74-8e58-a8842d6909d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:53.459030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.458976 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\" (UniqueName: \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\") pod \"cd752ac0-8178-4e74-8e58-a8842d6909d5\" (UID: \"cd752ac0-8178-4e74-8e58-a8842d6909d5\") " Oct 29 22:02:53.459104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.459044 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume" pod "816c43f4-5339-4fc1-92d4-9f15f53d5ac2" (UID: "816c43f4-5339-4fc1-92d4-9f15f53d5ac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:02:53.459895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.459866 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd752ac0-8178-4e74-8e58-a8842d6909d5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:53.460009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.459897 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:53.461240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.461213 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm" pod "816c43f4-5339-4fc1-92d4-9f15f53d5ac2" (UID: "816c43f4-5339-4fc1-92d4-9f15f53d5ac2"). InnerVolumeSpecName "kube-api-access-gjhkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:53.461336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.461245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj" pod "cd752ac0-8178-4e74-8e58-a8842d6909d5" (UID: "cd752ac0-8178-4e74-8e58-a8842d6909d5"). InnerVolumeSpecName "kube-api-access-4ljcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:02:53.561184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.561125 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhkm\" (UniqueName: \"kubernetes.io/projected/816c43f4-5339-4fc1-92d4-9f15f53d5ac2-kube-api-access-gjhkm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:53.561184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:53.561146 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljcj\" (UniqueName: \"kubernetes.io/projected/cd752ac0-8178-4e74-8e58-a8842d6909d5-kube-api-access-4ljcj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:02:54.037401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.037370 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b"] Oct 29 22:02:54.050123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.050093 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6a4236-pfq2b"] Oct 29 22:02:54.175603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.175579 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" Oct 29 22:02:54.175802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.175576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8" event={"ID":"cd752ac0-8178-4e74-8e58-a8842d6909d5","Type":"ContainerDied","Data":"426208f582646cb700a0e23185774267a6aa27c3eef4faa0b98f3264bbf6250e"} Oct 29 22:02:54.175802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.175709 4543 scope.go:122] "RemoveContainer" containerID="002e23bfa8761694cb858b785f8d2107e83dacc162900dc7116cac3c71d57b63" Oct 29 22:02:54.177198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.177177 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" event={"ID":"816c43f4-5339-4fc1-92d4-9f15f53d5ac2","Type":"ContainerDied","Data":"9d4804cdfc6f263a2434d12367f9f7ebab9f293247e9043ab315abc2bab8c042"} Oct 29 22:02:54.177306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.177197 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m" Oct 29 22:02:54.187647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.187624 4543 scope.go:122] "RemoveContainer" containerID="6af8eedd65e0764d9bcd17ba3d032c492e36ee072d99975c1541b7735507b2aa" Oct 29 22:02:54.236824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.236794 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:02:54.249779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.249680 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-579f09-7fffddd5ff-n8fz8"] Oct 29 22:02:54.286940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.286906 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:02:54.291559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.291529 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-579f09-bccb57fd4-rd55m"] Oct 29 22:02:54.388142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.388113 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3763038c-fb3b-43a6-b280-5d12cfee2052" path="/var/lib/kubelet/pods/3763038c-fb3b-43a6-b280-5d12cfee2052/volumes" Oct 29 22:02:54.388589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.388575 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" path="/var/lib/kubelet/pods/816c43f4-5339-4fc1-92d4-9f15f53d5ac2/volumes" Oct 29 22:02:54.388985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:02:54.388971 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd752ac0-8178-4e74-8e58-a8842d6909d5" path="/var/lib/kubelet/pods/cd752ac0-8178-4e74-8e58-a8842d6909d5/volumes" Oct 29 22:02:57.963282 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:02:57.963244 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:03:00.709649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.709612 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:03:00.710331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710315 4543 cpu_manager.go:414] "containerMap: removing container" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" Oct 29 22:03:00.710384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710332 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" Oct 29 22:03:00.710384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710367 4543 cpu_manager.go:414] "containerMap: removing container" podUID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" containerName="payload-processing" Oct 29 22:03:00.710384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710372 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" containerName="payload-processing" Oct 29 22:03:00.710384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710383 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710389 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710395 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710399 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710413 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cd752ac0-8178-4e74-8e58-a8842d6909d5" containerName="payload-pre-processing" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710417 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd752ac0-8178-4e74-8e58-a8842d6909d5" containerName="payload-pre-processing" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710431 4543 cpu_manager.go:414] "containerMap: removing container" podUID="55cfe408-d146-4ca9-a29d-b23850eb6cab" containerName="revoke-keys" Oct 29 22:03:00.710504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710438 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55cfe408-d146-4ca9-a29d-b23850eb6cab" containerName="revoke-keys" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710527 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710537 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd17290b-d871-4cd2-9843-198f3d1dfe68" containerName="istio-proxy" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710545 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd752ac0-8178-4e74-8e58-a8842d6909d5" containerName="payload-pre-processing" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710552 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="55cfe408-d146-4ca9-a29d-b23850eb6cab" containerName="revoke-keys" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710560 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="214d1999-43cc-48c3-aee6-b9210eaea845" containerName="payload-pre-processing" Oct 29 22:03:00.710705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.710569 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="816c43f4-5339-4fc1-92d4-9f15f53d5ac2" containerName="payload-processing" Oct 29 22:03:00.716099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.716056 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.735137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.735095 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:03:00.743377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.743341 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8867e7d\"" Oct 29 22:03:00.743500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.743455 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8867e7d-dockercfg-84rrf\"" Oct 29 22:03:00.769233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.769205 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8fr\" (UniqueName: \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.769497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.769478 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.808349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.808324 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:03:00.812988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.812963 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.887269 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jcp\" (UniqueName: \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.887359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.887464 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8fr\" (UniqueName: \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.887607 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.888700 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.906102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.896119 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:03:00.995901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.986933 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8fr\" (UniqueName: \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\") pod \"payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:00.995901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.991277 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jcp\" (UniqueName: \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:00.995901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.991552 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:00.997401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:00.997364 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:01.033436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:01.033398 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:01.050621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:01.050589 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jcp\" (UniqueName: \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\") pod \"payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:01.155885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:01.146533 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:01.385132 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:01.385103 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c8f9a1_1bbf_4d03_a20c_912c56fab678.slice/crio-b14bfb4aa7bc804ba123baaeb13d4c684899e5fb73b48d3407ea863df5bc7673 WatchSource:0}: Error finding container b14bfb4aa7bc804ba123baaeb13d4c684899e5fb73b48d3407ea863df5bc7673: Status 404 returned error can't find the container with id b14bfb4aa7bc804ba123baaeb13d4c684899e5fb73b48d3407ea863df5bc7673 Oct 29 22:03:01.385550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:01.385529 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:03:01.486364 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:01.486339 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca029fc_3d08_4213_94d1_3636143366b1.slice/crio-184dd41fdc9f62e06ba42e74997aa9284371f371fcdef8f6fa18e63eeab0cd93 WatchSource:0}: Error finding container 184dd41fdc9f62e06ba42e74997aa9284371f371fcdef8f6fa18e63eeab0cd93: Status 404 returned error can't find the container with id 184dd41fdc9f62e06ba42e74997aa9284371f371fcdef8f6fa18e63eeab0cd93 Oct 29 22:03:01.486702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:01.486680 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:03:02.262898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.262866 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" event={"ID":"aca029fc-3d08-4213-94d1-3636143366b1","Type":"ContainerStarted","Data":"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939"} Oct 29 22:03:02.263439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.262919 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" event={"ID":"aca029fc-3d08-4213-94d1-3636143366b1","Type":"ContainerStarted","Data":"184dd41fdc9f62e06ba42e74997aa9284371f371fcdef8f6fa18e63eeab0cd93"} Oct 29 22:03:02.263439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.262946 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:02.264802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.264768 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" event={"ID":"c7c8f9a1-1bbf-4d03-a20c-912c56fab678","Type":"ContainerStarted","Data":"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93"} Oct 29 22:03:02.264802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.264806 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" event={"ID":"c7c8f9a1-1bbf-4d03-a20c-912c56fab678","Type":"ContainerStarted","Data":"b14bfb4aa7bc804ba123baaeb13d4c684899e5fb73b48d3407ea863df5bc7673"} Oct 29 22:03:02.264985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.264856 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:02.308827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.308770 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" podStartSLOduration=2.308754525 podStartE2EDuration="2.308754525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:02.307962781 +0000 UTC m=+1244.049893454" watchObservedRunningTime="2026-10-29 22:03:02.308754525 +0000 UTC m=+1244.050685199" Oct 29 22:03:02.356724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:02.356682 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" podStartSLOduration=2.356670092 podStartE2EDuration="2.356670092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:02.352638268 +0000 UTC m=+1244.094568941" watchObservedRunningTime="2026-10-29 22:03:02.356670092 +0000 UTC m=+1244.098600764" Oct 29 22:03:03.046930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:03.046899 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv"] Oct 29 22:03:03.060012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:03.059985 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489e3b3-nxjrv"] Oct 29 22:03:04.386320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:04.386286 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2840dec5-b892-4231-a7f6-d5599d4d5a34" path="/var/lib/kubelet/pods/2840dec5-b892-4231-a7f6-d5599d4d5a34/volumes" Oct 29 22:03:06.492281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.492244 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:03:06.498402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.498376 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.550620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.547675 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:03:06.626200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.614426 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ad8e69-openshift-default-dockercfg-jblqd\"" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.632330 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633030 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633185 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633289 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633340 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633382 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8lc\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633452 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633482 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.636042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.633515 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735453 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8lc\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735527 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735547 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735567 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735812 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.735919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735883 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.735946 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736059 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736099 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736241 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736436 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.736550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.736531 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.738357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.738330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.738487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.738471 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.761477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.761425 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.761588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.761571 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8lc\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\") pod \"e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:06.944355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:06.944308 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:07.303399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:07.303372 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:03:07.303783 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:07.303755 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6665a2_7cb2_49dc_9a04_dccd722eccb1.slice/crio-554115ece6ba49e1770015ed1d393e7ee597344e81b7b49783db6709fb200b7c WatchSource:0}: Error finding container 554115ece6ba49e1770015ed1d393e7ee597344e81b7b49783db6709fb200b7c: Status 404 returned error can't find the container with id 554115ece6ba49e1770015ed1d393e7ee597344e81b7b49783db6709fb200b7c Oct 29 22:03:07.306678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:07.306633 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:07.306767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:07.306724 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:07.306767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:07.306766 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:08.318372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:08.318328 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerStarted","Data":"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa"} Oct 29 22:03:08.318827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:08.318404 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerStarted","Data":"554115ece6ba49e1770015ed1d393e7ee597344e81b7b49783db6709fb200b7c"} Oct 29 22:03:08.388459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:08.388411 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podStartSLOduration=2.388395874 podStartE2EDuration="2.388395874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:08.381561403 +0000 UTC m=+1250.123492090" watchObservedRunningTime="2026-10-29 22:03:08.388395874 +0000 UTC m=+1250.130326604" Oct 29 22:03:08.945373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:08.945070 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:08.951034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:08.951009 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:09.038512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:09.038485 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s"] Oct 29 22:03:09.042919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:09.042897 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c650fe-rvv9s"] Oct 29 22:03:09.333975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:09.333845 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:09.336220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:09.336191 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:03:10.386777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:10.386744 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd83061-3276-4968-996f-efe710794e22" path="/var/lib/kubelet/pods/1cd83061-3276-4968-996f-efe710794e22/volumes" Oct 29 22:03:13.109419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.109390 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:03:13.115155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.115134 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.142957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.142921 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:03:13.148198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.148175 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad8e69\"" Oct 29 22:03:13.148322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.148309 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad8e69-dockercfg-cjrsx\"" Oct 29 22:03:13.208413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.208386 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:03:13.212973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.212954 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.243949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.243924 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:03:13.270663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.270628 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.271039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.270709 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tmc\" (UniqueName: \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.281027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.280999 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:03:13.413910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.413827 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:03:13.466101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.456576 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.466101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.456646 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tmc\" (UniqueName: \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.472984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.472005 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.518087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.518034 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tmc\" (UniqueName: \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\") pod \"payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.559750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.559706 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct56s\" (UniqueName: \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.559960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.559940 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.661481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.661437 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct56s\" (UniqueName: \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.661694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.661675 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.662669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.662639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.687881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.687816 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct56s\" (UniqueName: \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\") pod \"payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:13.730340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.730309 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:13.827356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:13.827328 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:14.044241 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:14.044209 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d39ff70_9797_4553_a1bd_03edf17a4fcd.slice/crio-bbe9d3aa37f8d024bad32023b1e479b792cf80edf137d27a0a5b768b1ddc6b6a WatchSource:0}: Error finding container bbe9d3aa37f8d024bad32023b1e479b792cf80edf137d27a0a5b768b1ddc6b6a: Status 404 returned error can't find the container with id bbe9d3aa37f8d024bad32023b1e479b792cf80edf137d27a0a5b768b1ddc6b6a Oct 29 22:03:14.044973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.044952 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:03:14.150280 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:14.150253 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00668c48_f9af_4b13_809e_68850c4c7642.slice/crio-175a5fb666e237add257d90ed5b83e00d19b6d622f165e1f6b9d3feb32da9ca7 WatchSource:0}: Error finding container 175a5fb666e237add257d90ed5b83e00d19b6d622f165e1f6b9d3feb32da9ca7: Status 404 returned error can't find the container with id 175a5fb666e237add257d90ed5b83e00d19b6d622f165e1f6b9d3feb32da9ca7 Oct 29 22:03:14.151469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.151448 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:03:14.467619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.467584 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" event={"ID":"00668c48-f9af-4b13-809e-68850c4c7642","Type":"ContainerStarted","Data":"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577"} Oct 29 22:03:14.467806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.467649 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:14.467806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.467670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" event={"ID":"00668c48-f9af-4b13-809e-68850c4c7642","Type":"ContainerStarted","Data":"175a5fb666e237add257d90ed5b83e00d19b6d622f165e1f6b9d3feb32da9ca7"} Oct 29 22:03:14.469683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.469652 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" event={"ID":"5d39ff70-9797-4553-a1bd-03edf17a4fcd","Type":"ContainerStarted","Data":"d1e294183b83545350490da866d5bbef8ed1ed0f49f400073908527d1ce5570a"} Oct 29 22:03:14.469830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.469703 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:14.469830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.469727 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" event={"ID":"5d39ff70-9797-4553-a1bd-03edf17a4fcd","Type":"ContainerStarted","Data":"bbe9d3aa37f8d024bad32023b1e479b792cf80edf137d27a0a5b768b1ddc6b6a"} Oct 29 22:03:14.515575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.515535 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" podStartSLOduration=1.5155234069999999 podStartE2EDuration="1.515523407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:14.511940559 +0000 UTC m=+1256.253871233" watchObservedRunningTime="2026-10-29 22:03:14.515523407 +0000 UTC m=+1256.257454080" Oct 29 22:03:14.560181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:14.560140 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" podStartSLOduration=1.560124578 podStartE2EDuration="1.560124578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:14.555757229 +0000 UTC m=+1256.297687904" watchObservedRunningTime="2026-10-29 22:03:14.560124578 +0000 UTC m=+1256.302055251" Oct 29 22:03:16.278111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.278066 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:03:16.282929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.282908 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.304844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.304820 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:03:16.307505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.307136 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.307505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.307297 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8x2\" (UniqueName: \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.311736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.311712 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-442dc2-dockercfg-pbglt\"" Oct 29 22:03:16.311855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.311835 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-442dc2\"" Oct 29 22:03:16.475319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.460902 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.475319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.461113 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8x2\" (UniqueName: \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.475319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.462637 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.513065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.509789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8x2\" (UniqueName: \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\") pod \"payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.521722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.521695 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:03:16.565283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.565199 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:03:16.565821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.565467 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.600338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.600308 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:16.667998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.667914 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.669328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.669217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfssd\" (UniqueName: \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.773119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.773036 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.773309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.773125 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfssd\" (UniqueName: \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.774441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.774413 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.799897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.799864 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfssd\" (UniqueName: \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\") pod \"payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.908589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.908497 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:16.949860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:16.949822 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:03:16.953507 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:16.953480 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c87c450_426c_4ede_84a6_6b839f53c082.slice/crio-95541d62d885c4d6b9e6273d4529f114c3715f5767c91736c8004538ec22b93b WatchSource:0}: Error finding container 95541d62d885c4d6b9e6273d4529f114c3715f5767c91736c8004538ec22b93b: Status 404 returned error can't find the container with id 95541d62d885c4d6b9e6273d4529f114c3715f5767c91736c8004538ec22b93b Oct 29 22:03:17.226771 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:17.226744 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42eceb4_ce3f_4d75_83d8_53d2ae72b40c.slice/crio-29ee59d40fcb782cce9dd8f1d09a90d9ccb7a9b30a349ba2b0c4ee1cedcf2019 WatchSource:0}: Error finding container 29ee59d40fcb782cce9dd8f1d09a90d9ccb7a9b30a349ba2b0c4ee1cedcf2019: Status 404 returned error can't find the container with id 29ee59d40fcb782cce9dd8f1d09a90d9ccb7a9b30a349ba2b0c4ee1cedcf2019 Oct 29 22:03:17.227089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.227055 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:03:17.531322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.531233 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" event={"ID":"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c","Type":"ContainerStarted","Data":"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9"} Oct 29 22:03:17.531322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.531295 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:17.531322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.531317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" event={"ID":"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c","Type":"ContainerStarted","Data":"29ee59d40fcb782cce9dd8f1d09a90d9ccb7a9b30a349ba2b0c4ee1cedcf2019"} Oct 29 22:03:17.533137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.533105 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" event={"ID":"8c87c450-426c-4ede-84a6-6b839f53c082","Type":"ContainerStarted","Data":"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513"} Oct 29 22:03:17.533258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.533158 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" event={"ID":"8c87c450-426c-4ede-84a6-6b839f53c082","Type":"ContainerStarted","Data":"95541d62d885c4d6b9e6273d4529f114c3715f5767c91736c8004538ec22b93b"} Oct 29 22:03:17.533258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.533188 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:17.577307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.577263 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" podStartSLOduration=1.5772496 podStartE2EDuration="1.5772496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:17.574899082 +0000 UTC m=+1259.316829757" watchObservedRunningTime="2026-10-29 22:03:17.5772496 +0000 UTC m=+1259.319180272" Oct 29 22:03:17.626730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:17.626684 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" podStartSLOduration=1.62667106 podStartE2EDuration="1.62667106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:17.622632894 +0000 UTC m=+1259.364563569" watchObservedRunningTime="2026-10-29 22:03:17.62667106 +0000 UTC m=+1259.368601758" Oct 29 22:03:18.046938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:18.046907 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k"] Oct 29 22:03:18.051470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:18.051449 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e6a1bfb-kfd8k"] Oct 29 22:03:18.387492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:18.387407 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d7687c-36db-4e7a-bb3d-ba2f2648bba2" path="/var/lib/kubelet/pods/f3d7687c-36db-4e7a-bb3d-ba2f2648bba2/volumes" Oct 29 22:03:20.036421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:20.036388 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9"] Oct 29 22:03:20.049496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:20.049468 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eeeb8e0-9cvl9"] Oct 29 22:03:20.387380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:20.387308 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b575ec0-348c-4a3a-a701-3fcb70959a1b" path="/var/lib/kubelet/pods/0b575ec0-348c-4a3a-a701-3fcb70959a1b/volumes" Oct 29 22:03:25.481038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:25.481011 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:03:25.481428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:25.481091 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:03:27.228181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:27.228149 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g"] Oct 29 22:03:27.242233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:27.242193 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-579f09-zv42g"] Oct 29 22:03:27.962421 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:03:27.962386 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:03:28.419461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:28.419372 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55cfe408-d146-4ca9-a29d-b23850eb6cab" path="/var/lib/kubelet/pods/55cfe408-d146-4ca9-a29d-b23850eb6cab/volumes" Oct 29 22:03:28.544371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:28.544343 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:03:28.544528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:28.544418 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:03:36.091163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:36.091124 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd"] Oct 29 22:03:36.119100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:36.110656 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4da8fe8-wjnxd"] Oct 29 22:03:36.394950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:36.394872 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34264ca8-86fe-43d0-a53e-f68ee5aa3b81" path="/var/lib/kubelet/pods/34264ca8-86fe-43d0-a53e-f68ee5aa3b81/volumes" Oct 29 22:03:38.038966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:38.038934 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl"] Oct 29 22:03:38.052103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:38.052063 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2812bacc-5z6bl"] Oct 29 22:03:38.395679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:38.395598 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0861bf56-903c-4b10-8f3c-f819b83d7dbf" path="/var/lib/kubelet/pods/0861bf56-903c-4b10-8f3c-f819b83d7dbf/volumes" Oct 29 22:03:39.684799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.684768 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859"] Oct 29 22:03:39.690392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.690374 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.708659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.708637 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859"] Oct 29 22:03:39.717181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.717156 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:03:39.717181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.717180 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:03:39.717394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.717281 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:03:39.720620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.720596 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:03:39.802746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.802715 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.802880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.802832 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gnd\" (UniqueName: \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.803141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.803119 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.904705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.904681 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.904818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.904776 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.904869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.904821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gnd\" (UniqueName: \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.905472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.905446 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.907244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.907218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:39.930832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:39.930804 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gnd\" (UniqueName: \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\") pod \"maas-api-revoke-keys-e2e-api-a-fcce6b-h9859\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:40.005779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.005754 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:40.316915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.316839 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859"] Oct 29 22:03:40.588138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.588057 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:03:40.594623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.594595 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.610844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.610733 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:03:40.620024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.620002 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4d79fa-openshift-default-dockercfg-h57br\"" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.719482 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.719735 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.719786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.719837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.719910 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dr5l\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.720009 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.720103 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.720140 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.724103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.720309 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834436 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dr5l\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834525 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834607 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834716 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.834798 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.835119 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.835170 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.835198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.836135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.836062 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.839751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.839413 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.845693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.842183 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.845693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.842834 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.845693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.843103 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.845693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.843333 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.845693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.843554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.917100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.913360 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dr5l\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.917100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.915797 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 22:03:40.917100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.916024 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" containerID="cri-o://d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5" gracePeriod=30 Oct 29 22:03:40.928327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.924388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\") pod \"e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:40.928327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:40.925427 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" event={"ID":"8fadf590-bed5-4d97-b17a-d8e5c247b180","Type":"ContainerStarted","Data":"6f18bedaa7308377e52d6f9fe9ba31c8eed9eb3adce1074a2d8ec96c8250086c"} Oct 29 22:03:41.217012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.216977 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:41.588716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.588692 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:03:41.609007 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:41.608974 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b992505_27ea_4e99_8e66_4fabfd4e5522.slice/crio-ad9394fe84bfe41cf6a39ed5e2d704be710ac78ea8cd183bb2bf32d2b90fe84e WatchSource:0}: Error finding container ad9394fe84bfe41cf6a39ed5e2d704be710ac78ea8cd183bb2bf32d2b90fe84e: Status 404 returned error can't find the container with id ad9394fe84bfe41cf6a39ed5e2d704be710ac78ea8cd183bb2bf32d2b90fe84e Oct 29 22:03:41.611988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.611956 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:41.612069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.612024 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:41.612069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.612061 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:03:41.946554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.946461 4543 generic.go:356] "Generic (PLEG): container finished" podID="8fadf590-bed5-4d97-b17a-d8e5c247b180" containerID="6ee7e9a0d4f913e65dc520c8bfed6d2aec7a93b3d30a4ff2c63a3dc099e14290" exitCode=0 Oct 29 22:03:41.948748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.947601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" event={"ID":"8fadf590-bed5-4d97-b17a-d8e5c247b180","Type":"ContainerDied","Data":"6ee7e9a0d4f913e65dc520c8bfed6d2aec7a93b3d30a4ff2c63a3dc099e14290"} Oct 29 22:03:41.950536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.950504 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerStarted","Data":"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73"} Oct 29 22:03:41.950659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:41.950554 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerStarted","Data":"ad9394fe84bfe41cf6a39ed5e2d704be710ac78ea8cd183bb2bf32d2b90fe84e"} Oct 29 22:03:42.071271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:42.071221 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podStartSLOduration=2.071206452 podStartE2EDuration="2.071206452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:42.066854239 +0000 UTC m=+1283.808784924" watchObservedRunningTime="2026-10-29 22:03:42.071206452 +0000 UTC m=+1283.813137124" Oct 29 22:03:42.217742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:42.217714 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:42.223456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:42.223434 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:42.959647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:42.959613 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:42.961142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:42.961110 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:03:43.204091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.204051 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:43.280036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.280005 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\") pod \"8fadf590-bed5-4d97-b17a-d8e5c247b180\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " Oct 29 22:03:43.280328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.280306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\") pod \"8fadf590-bed5-4d97-b17a-d8e5c247b180\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " Oct 29 22:03:43.280420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.280363 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca" pod "8fadf590-bed5-4d97-b17a-d8e5c247b180" (UID: "8fadf590-bed5-4d97-b17a-d8e5c247b180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:43.280420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.280400 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\" (UniqueName: \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\") pod \"8fadf590-bed5-4d97-b17a-d8e5c247b180\" (UID: \"8fadf590-bed5-4d97-b17a-d8e5c247b180\") " Oct 29 22:03:43.281303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.281281 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:43.282424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.282402 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca" pod "8fadf590-bed5-4d97-b17a-d8e5c247b180" (UID: "8fadf590-bed5-4d97-b17a-d8e5c247b180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:03:43.282543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.282519 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd" pod "8fadf590-bed5-4d97-b17a-d8e5c247b180" (UID: "8fadf590-bed5-4d97-b17a-d8e5c247b180"). InnerVolumeSpecName "kube-api-access-c8gnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:43.440115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.431778 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fadf590-bed5-4d97-b17a-d8e5c247b180-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:43.440115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.431814 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gnd\" (UniqueName: \"kubernetes.io/projected/8fadf590-bed5-4d97-b17a-d8e5c247b180-kube-api-access-c8gnd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:43.970353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.970305 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" Oct 29 22:03:43.970353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.970317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859" event={"ID":"8fadf590-bed5-4d97-b17a-d8e5c247b180","Type":"ContainerDied","Data":"6f18bedaa7308377e52d6f9fe9ba31c8eed9eb3adce1074a2d8ec96c8250086c"} Oct 29 22:03:43.970807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:43.970416 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f18bedaa7308377e52d6f9fe9ba31c8eed9eb3adce1074a2d8ec96c8250086c" Oct 29 22:03:44.272231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.272045 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:03:44.273069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.273004 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8fadf590-bed5-4d97-b17a-d8e5c247b180" containerName="revoke-keys" Oct 29 22:03:44.273069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.273027 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fadf590-bed5-4d97-b17a-d8e5c247b180" containerName="revoke-keys" Oct 29 22:03:44.273490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.273208 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fadf590-bed5-4d97-b17a-d8e5c247b180" containerName="revoke-keys" Oct 29 22:03:44.277710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.277688 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.295193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.295171 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fe45b\"" Oct 29 22:03:44.298392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.298370 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fe45b-dockercfg-2mcsc\"" Oct 29 22:03:44.300129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.300107 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:03:44.349124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.349043 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.349327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.349172 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjbc\" (UniqueName: \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.527153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.500262 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.527153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.500384 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjbc\" (UniqueName: \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.527153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.501952 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.629157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.626354 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjbc\" (UniqueName: \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\") pod \"payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:44.895155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:44.894624 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:45.297267 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:45.297234 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69af68f_63ef_4740_b446_f2fc3c19f7b6.slice/crio-6dbb54bc56375b67446039811e24e1e177c4cada8bfadaed0ff12f6dfcce2950 WatchSource:0}: Error finding container 6dbb54bc56375b67446039811e24e1e177c4cada8bfadaed0ff12f6dfcce2950: Status 404 returned error can't find the container with id 6dbb54bc56375b67446039811e24e1e177c4cada8bfadaed0ff12f6dfcce2950 Oct 29 22:03:45.297606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:45.297446 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:03:45.993560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:45.993522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" event={"ID":"d69af68f-63ef-4740-b446-f2fc3c19f7b6","Type":"ContainerStarted","Data":"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d"} Oct 29 22:03:45.993697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:45.993579 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" event={"ID":"d69af68f-63ef-4740-b446-f2fc3c19f7b6","Type":"ContainerStarted","Data":"6dbb54bc56375b67446039811e24e1e177c4cada8bfadaed0ff12f6dfcce2950"} Oct 29 22:03:45.993697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:45.993605 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:46.050883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.050824 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" podStartSLOduration=2.050803728 podStartE2EDuration="2.050803728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:46.038199872 +0000 UTC m=+1287.780130545" watchObservedRunningTime="2026-10-29 22:03:46.050803728 +0000 UTC m=+1287.792734416" Oct 29 22:03:46.279826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.279794 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 22:03:46.344926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.344896 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.344983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345028 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345130 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345193 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345287 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345353 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345404 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:46.345778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345501 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\") pod \"037ce790-2dca-46d1-89a0-48fb94a8c693\" (UID: \"037ce790-2dca-46d1-89a0-48fb94a8c693\") " Oct 29 22:03:46.345778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345537 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:46.345778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:46.345980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.345807 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:46.346541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.346515 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.346665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.346544 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.346665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.346560 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.346665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.346574 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037ce790-2dca-46d1-89a0-48fb94a8c693-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.346665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.346587 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.347809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.347781 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:46.347916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.347890 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:03:46.347962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.347945 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:46.348559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.348543 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq" pod "037ce790-2dca-46d1-89a0-48fb94a8c693" (UID: "037ce790-2dca-46d1-89a0-48fb94a8c693"). InnerVolumeSpecName "kube-api-access-ngqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:46.448871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.448826 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.448871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.448858 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.448871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.448873 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqsq\" (UniqueName: \"kubernetes.io/projected/037ce790-2dca-46d1-89a0-48fb94a8c693-kube-api-access-ngqsq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:46.452117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:46.448890 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037ce790-2dca-46d1-89a0-48fb94a8c693-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.010943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.010909 4543 generic.go:356] "Generic (PLEG): container finished" podID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerID="d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5" exitCode=0 Oct 29 22:03:47.011240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.010956 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" Oct 29 22:03:47.011240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.010986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" event={"ID":"037ce790-2dca-46d1-89a0-48fb94a8c693","Type":"ContainerDied","Data":"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5"} Oct 29 22:03:47.011240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.011045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck" event={"ID":"037ce790-2dca-46d1-89a0-48fb94a8c693","Type":"ContainerDied","Data":"e8b706fb33620d0ae838580b80a7ebecc519ed3965b3cafb8d4dd83bc292e4d8"} Oct 29 22:03:47.011240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.011119 4543 scope.go:122] "RemoveContainer" containerID="d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5" Oct 29 22:03:47.022683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.022664 4543 scope.go:122] "RemoveContainer" containerID="d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5" Oct 29 22:03:47.022971 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:03:47.022950 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5\": container with ID starting with d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5 not found: ID does not exist" containerID="d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5" Oct 29 22:03:47.023050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.022978 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5"} err="failed to get container status \"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5\": rpc error: code = NotFound desc = could not find container \"d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5\": container with ID starting with d0abe6e6be92e1d320b1dcae3f60025fc2aaebb1cb72877fd8618d1d58554fe5 not found: ID does not exist" Oct 29 22:03:47.258239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.258152 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 22:03:47.297179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.285493 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-ddvck"] Oct 29 22:03:47.840364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.840332 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:03:47.841199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.841179 4543 cpu_manager.go:414] "containerMap: removing container" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" Oct 29 22:03:47.841252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.841202 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" Oct 29 22:03:47.841339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.841330 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" containerName="istio-proxy" Oct 29 22:03:47.846375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.846358 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:47.867006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.866949 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4d79fa\"" Oct 29 22:03:47.869415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.869392 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:03:47.873727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.873706 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4d79fa-dockercfg-9r22j\"" Oct 29 22:03:47.910629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.910578 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qwr\" (UniqueName: \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:47.910813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.910708 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:47.925836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.925795 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:03:48.012103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:47.999361 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.040098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.040043 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:03:48.051960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.047682 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qwr\" (UniqueName: \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:48.051960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.047849 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:48.051960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.049186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:48.059991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.059966 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn"] Oct 29 22:03:48.085101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.081012 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b4a526-xw2rn"] Oct 29 22:03:48.108193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.100816 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qwr\" (UniqueName: \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\") pod \"payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:48.149843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.149801 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.150088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.150053 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdg2\" (UniqueName: \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.167842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.167811 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:48.252057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.252026 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdg2\" (UniqueName: \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.252344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.252322 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.253783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.253758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.279505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.279479 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdg2\" (UniqueName: \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\") pod \"payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.356360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.356053 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:48.464064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.463998 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037ce790-2dca-46d1-89a0-48fb94a8c693" path="/var/lib/kubelet/pods/037ce790-2dca-46d1-89a0-48fb94a8c693/volumes" Oct 29 22:03:48.465114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.465051 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca31c8b2-b8ff-4430-b9c6-9df69b4e9854" path="/var/lib/kubelet/pods/ca31c8b2-b8ff-4430-b9c6-9df69b4e9854/volumes" Oct 29 22:03:48.492915 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:48.492878 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4521277a_d81d_4450_a77d_d5139599f2fc.slice/crio-91979b1c6ef13a415efb9efa9f7a378284a9e0fa1da9678fa9a78af272175c73 WatchSource:0}: Error finding container 91979b1c6ef13a415efb9efa9f7a378284a9e0fa1da9678fa9a78af272175c73: Status 404 returned error can't find the container with id 91979b1c6ef13a415efb9efa9f7a378284a9e0fa1da9678fa9a78af272175c73 Oct 29 22:03:48.494853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.494825 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:03:48.771734 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:48.771704 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45456c4c_6916_4832_94e5_d4f5e7aa8192.slice/crio-a40a66f11289ca7bbf70a85ff88366f1f518f328a3dce9d4f25e4b4d433e5264 WatchSource:0}: Error finding container a40a66f11289ca7bbf70a85ff88366f1f518f328a3dce9d4f25e4b4d433e5264: Status 404 returned error can't find the container with id a40a66f11289ca7bbf70a85ff88366f1f518f328a3dce9d4f25e4b4d433e5264 Oct 29 22:03:48.772338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:48.772315 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:03:49.082285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.082252 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" event={"ID":"45456c4c-6916-4832-94e5-d4f5e7aa8192","Type":"ContainerStarted","Data":"080ef44b52e9024fa7417a15be5ff5ec4f8611a5e3938943b2ba9a52a271ad93"} Oct 29 22:03:49.082688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.082313 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" event={"ID":"45456c4c-6916-4832-94e5-d4f5e7aa8192","Type":"ContainerStarted","Data":"a40a66f11289ca7bbf70a85ff88366f1f518f328a3dce9d4f25e4b4d433e5264"} Oct 29 22:03:49.082688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.082347 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:03:49.084129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.084095 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" event={"ID":"4521277a-d81d-4450-a77d-d5139599f2fc","Type":"ContainerStarted","Data":"4db757d4bc38a242baf74ed0c81c5b3fc65c55f32dbb5caf2408e29e12b41758"} Oct 29 22:03:49.084242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.084156 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:03:49.084242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.084182 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" event={"ID":"4521277a-d81d-4450-a77d-d5139599f2fc","Type":"ContainerStarted","Data":"91979b1c6ef13a415efb9efa9f7a378284a9e0fa1da9678fa9a78af272175c73"} Oct 29 22:03:49.128174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.128128 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" podStartSLOduration=2.128115893 podStartE2EDuration="2.128115893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:49.125641204 +0000 UTC m=+1290.867571879" watchObservedRunningTime="2026-10-29 22:03:49.128115893 +0000 UTC m=+1290.870046563" Oct 29 22:03:49.171389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:49.171345 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" podStartSLOduration=2.1713314439999998 podStartE2EDuration="2.171331444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:49.169089847 +0000 UTC m=+1290.911020521" watchObservedRunningTime="2026-10-29 22:03:49.171331444 +0000 UTC m=+1290.913262117" Oct 29 22:03:51.250826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:51.250796 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:03:51.286661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:51.286633 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:03:56.251419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.251376 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" podUID="25d1a98f-7443-491c-bf7b-e50d45255017" containerName="payload-processing" containerID="cri-o://7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1" gracePeriod=25 Oct 29 22:03:56.287292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.287261 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" podUID="36f36a6c-459b-4d2a-94b5-25e28718f56b" containerName="payload-pre-processing" containerID="cri-o://606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536" gracePeriod=25 Oct 29 22:03:56.595033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.595005 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:03:56.631334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.631312 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:03:56.692518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.691036 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\") pod \"25d1a98f-7443-491c-bf7b-e50d45255017\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " Oct 29 22:03:56.692518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.691414 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\" (UniqueName: \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\") pod \"25d1a98f-7443-491c-bf7b-e50d45255017\" (UID: \"25d1a98f-7443-491c-bf7b-e50d45255017\") " Oct 29 22:03:56.692891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.692865 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume" pod "25d1a98f-7443-491c-bf7b-e50d45255017" (UID: "25d1a98f-7443-491c-bf7b-e50d45255017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:56.693755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.693732 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1a98f-7443-491c-bf7b-e50d45255017-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:56.705995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.705968 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8" pod "25d1a98f-7443-491c-bf7b-e50d45255017" (UID: "25d1a98f-7443-491c-bf7b-e50d45255017"). InnerVolumeSpecName "kube-api-access-kj2x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:56.794906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.794827 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\" (UniqueName: \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\") pod \"36f36a6c-459b-4d2a-94b5-25e28718f56b\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " Oct 29 22:03:56.795037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.795012 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\") pod \"36f36a6c-459b-4d2a-94b5-25e28718f56b\" (UID: \"36f36a6c-459b-4d2a-94b5-25e28718f56b\") " Oct 29 22:03:56.795439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.795414 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume" pod "36f36a6c-459b-4d2a-94b5-25e28718f56b" (UID: "36f36a6c-459b-4d2a-94b5-25e28718f56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:56.795603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.795590 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f36a6c-459b-4d2a-94b5-25e28718f56b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:56.795642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.795606 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2x8\" (UniqueName: \"kubernetes.io/projected/25d1a98f-7443-491c-bf7b-e50d45255017-kube-api-access-kj2x8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:56.797029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.797006 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l" pod "36f36a6c-459b-4d2a-94b5-25e28718f56b" (UID: "36f36a6c-459b-4d2a-94b5-25e28718f56b"). InnerVolumeSpecName "kube-api-access-m6w8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:56.896941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.896906 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6w8l\" (UniqueName: \"kubernetes.io/projected/36f36a6c-459b-4d2a-94b5-25e28718f56b-kube-api-access-m6w8l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:03:56.944002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.943975 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8"] Oct 29 22:03:56.944735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944720 4543 cpu_manager.go:414] "containerMap: removing container" podUID="25d1a98f-7443-491c-bf7b-e50d45255017" containerName="payload-processing" Oct 29 22:03:56.944785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944738 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d1a98f-7443-491c-bf7b-e50d45255017" containerName="payload-processing" Oct 29 22:03:56.944785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944757 4543 cpu_manager.go:414] "containerMap: removing container" podUID="36f36a6c-459b-4d2a-94b5-25e28718f56b" containerName="payload-pre-processing" Oct 29 22:03:56.944785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944772 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f36a6c-459b-4d2a-94b5-25e28718f56b" containerName="payload-pre-processing" Oct 29 22:03:56.944938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944929 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d1a98f-7443-491c-bf7b-e50d45255017" containerName="payload-processing" Oct 29 22:03:56.944986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.944951 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f36a6c-459b-4d2a-94b5-25e28718f56b" containerName="payload-pre-processing" Oct 29 22:03:56.948715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.948694 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:56.965950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.965931 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:03:56.966215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.966192 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:03:56.969135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.969114 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8"] Oct 29 22:03:56.975431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.975410 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:03:56.975536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:56.975441 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:03:57.014543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.014519 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:03:57.101820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.101743 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.101968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.101918 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.102051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.102035 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rd84\" (UniqueName: \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.203473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.203436 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.203671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.203648 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.203795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.203780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rd84\" (UniqueName: \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.204379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.204355 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.206065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.206033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.229609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.229590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rd84\" (UniqueName: \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\") pod \"maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.243626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.243597 4543 generic.go:356] "Generic (PLEG): container finished" podID="25d1a98f-7443-491c-bf7b-e50d45255017" containerID="7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1" exitCode=0 Oct 29 22:03:57.243714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.243652 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" Oct 29 22:03:57.243714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.243670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" event={"ID":"25d1a98f-7443-491c-bf7b-e50d45255017","Type":"ContainerDied","Data":"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1"} Oct 29 22:03:57.243816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.243741 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg" event={"ID":"25d1a98f-7443-491c-bf7b-e50d45255017","Type":"ContainerDied","Data":"8b92bbafa879c253bcbe6578a4059d26a823c253db71cdbe84c2176459fe1a3a"} Oct 29 22:03:57.243816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.243762 4543 scope.go:122] "RemoveContainer" containerID="7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1" Oct 29 22:03:57.245490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.245460 4543 generic.go:356] "Generic (PLEG): container finished" podID="36f36a6c-459b-4d2a-94b5-25e28718f56b" containerID="606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536" exitCode=0 Oct 29 22:03:57.245604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.245522 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" Oct 29 22:03:57.245604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.245564 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" event={"ID":"36f36a6c-459b-4d2a-94b5-25e28718f56b","Type":"ContainerDied","Data":"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536"} Oct 29 22:03:57.245718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.245605 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj" event={"ID":"36f36a6c-459b-4d2a-94b5-25e28718f56b","Type":"ContainerDied","Data":"81196075791e59c0af1d140bddab7dc7e38c0a1b78837802366582c3e9583fe4"} Oct 29 22:03:57.257479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.257306 4543 scope.go:122] "RemoveContainer" containerID="7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1" Oct 29 22:03:57.257789 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:03:57.257589 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1\": container with ID starting with 7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1 not found: ID does not exist" containerID="7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1" Oct 29 22:03:57.257789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.257620 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1"} err="failed to get container status \"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1\": rpc error: code = NotFound desc = could not find container \"7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1\": container with ID starting with 7ac5dfb048a7d9e9bcd6945586a450016219b8679feca5311d809767ecc8b4f1 not found: ID does not exist" Oct 29 22:03:57.257789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.257645 4543 scope.go:122] "RemoveContainer" containerID="606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536" Oct 29 22:03:57.264568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.264550 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:03:57.288114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.288058 4543 scope.go:122] "RemoveContainer" containerID="606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536" Oct 29 22:03:57.288474 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:03:57.288452 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536\": container with ID starting with 606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536 not found: ID does not exist" containerID="606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536" Oct 29 22:03:57.288558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.288485 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536"} err="failed to get container status \"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536\": rpc error: code = NotFound desc = could not find container \"606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536\": container with ID starting with 606fdbfd3a94bb355bf866a2a0fb30cd31071c6c5b180d99922e2547a753d536 not found: ID does not exist" Oct 29 22:03:57.307364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.307342 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:03:57.313185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.313158 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fcce6b-6b7868556c-frkrj"] Oct 29 22:03:57.355850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.353890 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:03:57.359694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.359667 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fcce6b-6f5455f5b-rfxsg"] Oct 29 22:03:57.591320 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:03:57.591291 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032689a2_2113_473f_b0b7_481f33968c89.slice/crio-c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76 WatchSource:0}: Error finding container c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76: Status 404 returned error can't find the container with id c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76 Oct 29 22:03:57.591926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:57.591881 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8"] Oct 29 22:03:57.967429 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:03:57.963678 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:03:58.256866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:58.256829 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" event={"ID":"032689a2-2113-473f-b0b7-481f33968c89","Type":"ContainerStarted","Data":"20ee25a42b92abcc0f544612e1aebb1c61a9dfe4853599d7519ea1ffac003b41"} Oct 29 22:03:58.257036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:58.256881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" event={"ID":"032689a2-2113-473f-b0b7-481f33968c89","Type":"ContainerStarted","Data":"c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76"} Oct 29 22:03:58.301629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:58.301575 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" podStartSLOduration=2.301561814 podStartE2EDuration="2.301561814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:58.300393641 +0000 UTC m=+1300.042324316" watchObservedRunningTime="2026-10-29 22:03:58.301561814 +0000 UTC m=+1300.043492545" Oct 29 22:03:58.387695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:58.387668 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d1a98f-7443-491c-bf7b-e50d45255017" path="/var/lib/kubelet/pods/25d1a98f-7443-491c-bf7b-e50d45255017/volumes" Oct 29 22:03:58.388142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:58.388128 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f36a6c-459b-4d2a-94b5-25e28718f56b" path="/var/lib/kubelet/pods/36f36a6c-459b-4d2a-94b5-25e28718f56b/volumes" Oct 29 22:03:59.267841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:59.267765 4543 generic.go:356] "Generic (PLEG): container finished" podID="032689a2-2113-473f-b0b7-481f33968c89" containerID="20ee25a42b92abcc0f544612e1aebb1c61a9dfe4853599d7519ea1ffac003b41" exitCode=0 Oct 29 22:03:59.268001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:03:59.267850 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" event={"ID":"032689a2-2113-473f-b0b7-481f33968c89","Type":"ContainerDied","Data":"20ee25a42b92abcc0f544612e1aebb1c61a9dfe4853599d7519ea1ffac003b41"} Oct 29 22:04:00.096016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.095978 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:04:00.096427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.096238 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:04:00.457668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.457647 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:04:00.568296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.568259 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\") pod \"032689a2-2113-473f-b0b7-481f33968c89\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " Oct 29 22:04:00.568494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.568472 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\" (UniqueName: \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\") pod \"032689a2-2113-473f-b0b7-481f33968c89\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " Oct 29 22:04:00.568760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.568698 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca" pod "032689a2-2113-473f-b0b7-481f33968c89" (UID: "032689a2-2113-473f-b0b7-481f33968c89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:00.568760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.568727 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\") pod \"032689a2-2113-473f-b0b7-481f33968c89\" (UID: \"032689a2-2113-473f-b0b7-481f33968c89\") " Oct 29 22:04:00.569518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.569486 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032689a2-2113-473f-b0b7-481f33968c89-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:00.575210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.574964 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca" pod "032689a2-2113-473f-b0b7-481f33968c89" (UID: "032689a2-2113-473f-b0b7-481f33968c89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:00.575210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.575064 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84" pod "032689a2-2113-473f-b0b7-481f33968c89" (UID: "032689a2-2113-473f-b0b7-481f33968c89"). InnerVolumeSpecName "kube-api-access-5rd84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:00.671017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.670980 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032689a2-2113-473f-b0b7-481f33968c89-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:00.671017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:00.671014 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rd84\" (UniqueName: \"kubernetes.io/projected/032689a2-2113-473f-b0b7-481f33968c89-kube-api-access-5rd84\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:01.287699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:01.287667 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" event={"ID":"032689a2-2113-473f-b0b7-481f33968c89","Type":"ContainerDied","Data":"c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76"} Oct 29 22:04:01.288091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:01.287713 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b867b95bf26e29346be368003bb21b84c9cf63c2f54641dd8573eea2d97e76" Oct 29 22:04:01.288091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:01.287678 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8" Oct 29 22:04:02.037526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:02.037488 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2"] Oct 29 22:04:02.050858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:02.050831 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b47e61-768z2"] Oct 29 22:04:02.394336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:02.394255 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e649504e-5f89-4ca5-98d7-38a9e6e4b8a8" path="/var/lib/kubelet/pods/e649504e-5f89-4ca5-98d7-38a9e6e4b8a8/volumes" Oct 29 22:04:07.495847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:07.495815 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:04:07.533086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:07.533043 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:04:09.038262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:09.038231 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk"] Oct 29 22:04:09.051426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:09.051398 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c85271-cfctk"] Oct 29 22:04:10.388275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:10.388246 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb36c573-bd84-41d1-a531-f25467e895a0" path="/var/lib/kubelet/pods/fb36c573-bd84-41d1-a531-f25467e895a0/volumes" Oct 29 22:04:12.496807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.496759 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" podUID="aca029fc-3d08-4213-94d1-3636143366b1" containerName="payload-processing" containerID="cri-o://28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939" gracePeriod=25 Oct 29 22:04:12.533486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.533444 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" podUID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" containerName="payload-pre-processing" containerID="cri-o://b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93" gracePeriod=25 Oct 29 22:04:12.897614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.897395 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:04:12.925264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.925238 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:04:12.972762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.972720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\") pod \"aca029fc-3d08-4213-94d1-3636143366b1\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " Oct 29 22:04:12.972953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.972869 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\" (UniqueName: \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\") pod \"aca029fc-3d08-4213-94d1-3636143366b1\" (UID: \"aca029fc-3d08-4213-94d1-3636143366b1\") " Oct 29 22:04:12.973057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.973037 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\" (UniqueName: \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\") pod \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " Oct 29 22:04:12.973153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.973093 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume" pod "aca029fc-3d08-4213-94d1-3636143366b1" (UID: "aca029fc-3d08-4213-94d1-3636143366b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:12.973796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.973769 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca029fc-3d08-4213-94d1-3636143366b1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:12.975395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.975367 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp" pod "aca029fc-3d08-4213-94d1-3636143366b1" (UID: "aca029fc-3d08-4213-94d1-3636143366b1"). InnerVolumeSpecName "kube-api-access-v2jcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:12.975820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:12.975799 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr" pod "c7c8f9a1-1bbf-4d03-a20c-912c56fab678" (UID: "c7c8f9a1-1bbf-4d03-a20c-912c56fab678"). InnerVolumeSpecName "kube-api-access-bp8fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:13.075140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.075006 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\") pod \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\" (UID: \"c7c8f9a1-1bbf-4d03-a20c-912c56fab678\") " Oct 29 22:04:13.075502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.075470 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume" pod "c7c8f9a1-1bbf-4d03-a20c-912c56fab678" (UID: "c7c8f9a1-1bbf-4d03-a20c-912c56fab678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:13.075710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.075690 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:13.075779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.075717 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jcp\" (UniqueName: \"kubernetes.io/projected/aca029fc-3d08-4213-94d1-3636143366b1-kube-api-access-v2jcp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:13.075779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.075730 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8fr\" (UniqueName: \"kubernetes.io/projected/c7c8f9a1-1bbf-4d03-a20c-912c56fab678-kube-api-access-bp8fr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:13.416110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.416007 4543 generic.go:356] "Generic (PLEG): container finished" podID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" containerID="b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93" exitCode=0 Oct 29 22:04:13.416110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.416048 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" Oct 29 22:04:13.416359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.416131 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" event={"ID":"c7c8f9a1-1bbf-4d03-a20c-912c56fab678","Type":"ContainerDied","Data":"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93"} Oct 29 22:04:13.416359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.416188 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9" event={"ID":"c7c8f9a1-1bbf-4d03-a20c-912c56fab678","Type":"ContainerDied","Data":"b14bfb4aa7bc804ba123baaeb13d4c684899e5fb73b48d3407ea863df5bc7673"} Oct 29 22:04:13.416359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.416215 4543 scope.go:122] "RemoveContainer" containerID="b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93" Oct 29 22:04:13.417795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.417772 4543 generic.go:356] "Generic (PLEG): container finished" podID="aca029fc-3d08-4213-94d1-3636143366b1" containerID="28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939" exitCode=0 Oct 29 22:04:13.417906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.417818 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" Oct 29 22:04:13.417906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.417876 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" event={"ID":"aca029fc-3d08-4213-94d1-3636143366b1","Type":"ContainerDied","Data":"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939"} Oct 29 22:04:13.417990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.417939 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj" event={"ID":"aca029fc-3d08-4213-94d1-3636143366b1","Type":"ContainerDied","Data":"184dd41fdc9f62e06ba42e74997aa9284371f371fcdef8f6fa18e63eeab0cd93"} Oct 29 22:04:13.428742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.428728 4543 scope.go:122] "RemoveContainer" containerID="b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93" Oct 29 22:04:13.429007 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:13.428982 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93\": container with ID starting with b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93 not found: ID does not exist" containerID="b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93" Oct 29 22:04:13.429065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.429010 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93"} err="failed to get container status \"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93\": rpc error: code = NotFound desc = could not find container \"b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93\": container with ID starting with b60097231749b26ab0753cc56238e243ccd0f85bebd5f4708226d9aacd38ba93 not found: ID does not exist" Oct 29 22:04:13.429065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.429031 4543 scope.go:122] "RemoveContainer" containerID="28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939" Oct 29 22:04:13.438981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.438966 4543 scope.go:122] "RemoveContainer" containerID="28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939" Oct 29 22:04:13.439262 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:13.439232 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939\": container with ID starting with 28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939 not found: ID does not exist" containerID="28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939" Oct 29 22:04:13.439314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.439262 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939"} err="failed to get container status \"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939\": rpc error: code = NotFound desc = could not find container \"28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939\": container with ID starting with 28eac00c8929ff6fb28f6d70c4b7ada7906a63080ffe6b0f4af065feaf9b1939 not found: ID does not exist" Oct 29 22:04:13.475316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.475292 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:04:13.488528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.488509 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8867e7d-69756875-bwkw9"] Oct 29 22:04:13.517599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.517579 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:04:13.522277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:13.522260 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8867e7d-55f67fcf98-bzdtj"] Oct 29 22:04:14.034583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:14.034555 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv"] Oct 29 22:04:14.047495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:14.047469 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b7f41-vzvmv"] Oct 29 22:04:14.446635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:14.446530 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8401a820-eded-4768-baca-88a723cfbb08" path="/var/lib/kubelet/pods/8401a820-eded-4768-baca-88a723cfbb08/volumes" Oct 29 22:04:14.447508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:14.447482 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca029fc-3d08-4213-94d1-3636143366b1" path="/var/lib/kubelet/pods/aca029fc-3d08-4213-94d1-3636143366b1/volumes" Oct 29 22:04:14.458096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:14.448284 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" path="/var/lib/kubelet/pods/c7c8f9a1-1bbf-4d03-a20c-912c56fab678/volumes" Oct 29 22:04:16.197614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.197581 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn"] Oct 29 22:04:16.198491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198472 4543 cpu_manager.go:414] "containerMap: removing container" podUID="032689a2-2113-473f-b0b7-481f33968c89" containerName="revoke-keys" Oct 29 22:04:16.198584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198493 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032689a2-2113-473f-b0b7-481f33968c89" containerName="revoke-keys" Oct 29 22:04:16.198584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198537 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" containerName="payload-pre-processing" Oct 29 22:04:16.198584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198542 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" containerName="payload-pre-processing" Oct 29 22:04:16.198584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198555 4543 cpu_manager.go:414] "containerMap: removing container" podUID="aca029fc-3d08-4213-94d1-3636143366b1" containerName="payload-processing" Oct 29 22:04:16.198584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198562 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca029fc-3d08-4213-94d1-3636143366b1" containerName="payload-processing" Oct 29 22:04:16.198798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198766 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca029fc-3d08-4213-94d1-3636143366b1" containerName="payload-processing" Oct 29 22:04:16.198798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198786 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="032689a2-2113-473f-b0b7-481f33968c89" containerName="revoke-keys" Oct 29 22:04:16.198864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.198804 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c8f9a1-1bbf-4d03-a20c-912c56fab678" containerName="payload-pre-processing" Oct 29 22:04:16.204512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.204495 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.220825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.220805 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn"] Oct 29 22:04:16.230586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.229670 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:04:16.230586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.229700 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:04:16.230586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.229720 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:04:16.230586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.229755 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:04:16.357790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.357765 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9kv\" (UniqueName: \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.357897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.357808 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.357897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.357878 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.459227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.459139 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9kv\" (UniqueName: \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.459227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.459206 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.459424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.459248 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.459959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.459941 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.462778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.462758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.485219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.485200 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9kv\" (UniqueName: \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\") pod \"maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.520028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.520009 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:16.830045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:16.830023 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn"] Oct 29 22:04:16.830924 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:04:16.830897 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a1364_f023_438f_93c9_d6ea4861032a.slice/crio-2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96 WatchSource:0}: Error finding container 2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96: Status 404 returned error can't find the container with id 2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96 Oct 29 22:04:17.511126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:17.511024 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" event={"ID":"867a1364-f023-438f-93c9-d6ea4861032a","Type":"ContainerStarted","Data":"572ace117f3789d4379d4131ecfdac5a450ecd4f1c907b553f4e2255956e1b9a"} Oct 29 22:04:17.511126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:17.511096 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" event={"ID":"867a1364-f023-438f-93c9-d6ea4861032a","Type":"ContainerStarted","Data":"2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96"} Oct 29 22:04:17.560268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:17.560204 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" podStartSLOduration=1.560185804 podStartE2EDuration="1.560185804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:17.559914258 +0000 UTC m=+1319.301844938" watchObservedRunningTime="2026-10-29 22:04:17.560185804 +0000 UTC m=+1319.302116478" Oct 29 22:04:18.212603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.208702 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 22:04:18.212603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.208959 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" containerID="cri-o://0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf" gracePeriod=30 Oct 29 22:04:18.250331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.250225 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 22:04:18.250543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.250507 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" containerID="cri-o://3aefc17de225d12a960947c2cbf2b745f1136c49265fa3a3ae40ee76776f0868" gracePeriod=30 Oct 29 22:04:18.259135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.258195 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 22:04:18.265821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.265724 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:04:18.266992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.266051 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" containerID="cri-o://68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7" gracePeriod=30 Oct 29 22:04:18.290126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.288171 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:04:18.521023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.520931 4543 generic.go:356] "Generic (PLEG): container finished" podID="867a1364-f023-438f-93c9-d6ea4861032a" containerID="572ace117f3789d4379d4131ecfdac5a450ecd4f1c907b553f4e2255956e1b9a" exitCode=0 Oct 29 22:04:18.521388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:18.521018 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" event={"ID":"867a1364-f023-438f-93c9-d6ea4861032a","Type":"ContainerDied","Data":"572ace117f3789d4379d4131ecfdac5a450ecd4f1c907b553f4e2255956e1b9a"} Oct 29 22:04:19.042401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.042339 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv"] Oct 29 22:04:19.048854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.048831 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ad6b62-kthdv"] Oct 29 22:04:19.071206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.071181 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd"] Oct 29 22:04:19.077301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.077268 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.095461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.095435 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd"] Oct 29 22:04:19.148554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.148527 4543 scope.go:122] "RemoveContainer" containerID="b02b80154c00993d06202dbb20e077ac93d536b40b3e17d721dc3fce0fce4c16" Oct 29 22:04:19.153587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.153559 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptjr\" (UniqueName: \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.153922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.153897 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.154039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.154021 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.160559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.160533 4543 scope.go:122] "RemoveContainer" containerID="406a8dc1b429eeaff172a1ab455943602b219d694d69ba29fc54769c537c3671" Oct 29 22:04:19.170756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.170740 4543 scope.go:122] "RemoveContainer" containerID="efea16e9c1c39e484d82a3819c9e972206fcbb4be0c1e02fb97f816d31b71f58" Oct 29 22:04:19.180033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.180017 4543 scope.go:122] "RemoveContainer" containerID="e928ec635483b3176b77f2cbcc49687439fee23f77f1759374c58b1bbbd3766a" Oct 29 22:04:19.190211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.190192 4543 scope.go:122] "RemoveContainer" containerID="b7bab7bf7f19dd0c68d49049761747b93e8d64bc1f0c6a19e06f11755ad65f76" Oct 29 22:04:19.204479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.202722 4543 scope.go:122] "RemoveContainer" containerID="0f85dcad1213621db552b158da3d7baa71a2857fa67189a0b4049068533e0bd0" Oct 29 22:04:19.259641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.258629 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.259641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.258708 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptjr\" (UniqueName: \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.259641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.259041 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.268109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.260737 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.333147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.322621 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.353174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.350784 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptjr\" (UniqueName: \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.369099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.359232 4543 scope.go:122] "RemoveContainer" containerID="429974dbc27f0a3ca8da94de7747205e7d3ff9acbc4863e2cdff61e5a3c1bfa9" Oct 29 22:04:19.403102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.397887 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:19.403102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.402906 4543 scope.go:122] "RemoveContainer" containerID="29f48d365a2495616c264cac5159748855f413ed9a9fecefea5d57b1c9fcf9e1" Oct 29 22:04:19.967887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:19.967861 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd"] Oct 29 22:04:19.968422 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:04:19.968401 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe1ba54_13d2_4a4e_a0ef_53683cae0487.slice/crio-79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a WatchSource:0}: Error finding container 79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a: Status 404 returned error can't find the container with id 79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a Oct 29 22:04:20.002719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.002698 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:20.084819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.084789 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\") pod \"867a1364-f023-438f-93c9-d6ea4861032a\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " Oct 29 22:04:20.084954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.084887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\") pod \"867a1364-f023-438f-93c9-d6ea4861032a\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " Oct 29 22:04:20.085035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.084998 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\" (UniqueName: \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\") pod \"867a1364-f023-438f-93c9-d6ea4861032a\" (UID: \"867a1364-f023-438f-93c9-d6ea4861032a\") " Oct 29 22:04:20.085213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.085190 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca" pod "867a1364-f023-438f-93c9-d6ea4861032a" (UID: "867a1364-f023-438f-93c9-d6ea4861032a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:20.085810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.085785 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:20.087165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.087138 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca" pod "867a1364-f023-438f-93c9-d6ea4861032a" (UID: "867a1364-f023-438f-93c9-d6ea4861032a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:20.087606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.087587 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv" pod "867a1364-f023-438f-93c9-d6ea4861032a" (UID: "867a1364-f023-438f-93c9-d6ea4861032a"). InnerVolumeSpecName "kube-api-access-pl9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:20.186847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.186814 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a1364-f023-438f-93c9-d6ea4861032a-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:20.186847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.186842 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9kv\" (UniqueName: \"kubernetes.io/projected/867a1364-f023-438f-93c9-d6ea4861032a-kube-api-access-pl9kv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:20.387847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.387808 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cced06e-25ae-439b-94b5-28d6f93d8c15" path="/var/lib/kubelet/pods/6cced06e-25ae-439b-94b5-28d6f93d8c15/volumes" Oct 29 22:04:20.784097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.779953 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" event={"ID":"867a1364-f023-438f-93c9-d6ea4861032a","Type":"ContainerDied","Data":"2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96"} Oct 29 22:04:20.784097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.780019 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e64ebaa88335496a2bbf8d88dc1c78bea1ed2bf1b365b5e9fe734a4f2c54a96" Oct 29 22:04:20.784097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.780123 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn" Oct 29 22:04:20.786628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.786591 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" event={"ID":"ffe1ba54-13d2-4a4e-a0ef-53683cae0487","Type":"ContainerStarted","Data":"b1e410f9353b9554e91490d5d53ccbe9838420a6f7907a04a8d82dea0c32b739"} Oct 29 22:04:20.786780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.786655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" event={"ID":"ffe1ba54-13d2-4a4e-a0ef-53683cae0487","Type":"ContainerStarted","Data":"79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a"} Oct 29 22:04:20.849730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:20.849647 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" podStartSLOduration=1.849618586 podStartE2EDuration="1.849618586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:20.842304709 +0000 UTC m=+1322.584235383" watchObservedRunningTime="2026-10-29 22:04:20.849618586 +0000 UTC m=+1322.591549295" Oct 29 22:04:21.802096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:21.801725 4543 generic.go:356] "Generic (PLEG): container finished" podID="ffe1ba54-13d2-4a4e-a0ef-53683cae0487" containerID="b1e410f9353b9554e91490d5d53ccbe9838420a6f7907a04a8d82dea0c32b739" exitCode=0 Oct 29 22:04:21.802096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:21.801842 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" event={"ID":"ffe1ba54-13d2-4a4e-a0ef-53683cae0487","Type":"ContainerDied","Data":"b1e410f9353b9554e91490d5d53ccbe9838420a6f7907a04a8d82dea0c32b739"} Oct 29 22:04:22.258539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:22.258504 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8"] Oct 29 22:04:22.263019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:22.262994 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8867e7d-xw8d8"] Oct 29 22:04:22.394351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:22.394319 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032689a2-2113-473f-b0b7-481f33968c89" path="/var/lib/kubelet/pods/032689a2-2113-473f-b0b7-481f33968c89/volumes" Oct 29 22:04:23.059762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.059732 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:23.176795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.176719 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\" (UniqueName: \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\") pod \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " Oct 29 22:04:23.176999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.176978 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\") pod \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " Oct 29 22:04:23.177101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.177049 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\") pod \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\" (UID: \"ffe1ba54-13d2-4a4e-a0ef-53683cae0487\") " Oct 29 22:04:23.178350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.178314 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca" pod "ffe1ba54-13d2-4a4e-a0ef-53683cae0487" (UID: "ffe1ba54-13d2-4a4e-a0ef-53683cae0487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:23.179782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.179756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca" pod "ffe1ba54-13d2-4a4e-a0ef-53683cae0487" (UID: "ffe1ba54-13d2-4a4e-a0ef-53683cae0487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:23.179930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.179904 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr" pod "ffe1ba54-13d2-4a4e-a0ef-53683cae0487" (UID: "ffe1ba54-13d2-4a4e-a0ef-53683cae0487"). InnerVolumeSpecName "kube-api-access-4ptjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:23.267005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.266970 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" podUID="00668c48-f9af-4b13-809e-68850c4c7642" containerName="payload-processing" containerID="cri-o://2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577" gracePeriod=25 Oct 29 22:04:23.279594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.279570 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.279594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.279596 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ptjr\" (UniqueName: \"kubernetes.io/projected/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-kube-api-access-4ptjr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.279718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.279606 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1ba54-13d2-4a4e-a0ef-53683cae0487-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.288902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.288854 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" podUID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" containerName="payload-pre-processing" containerID="cri-o://d1e294183b83545350490da866d5bbef8ed1ed0f49f400073908527d1ce5570a" gracePeriod=25 Oct 29 22:04:23.627948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.627915 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 22:04:23.686539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686414 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686484 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686653 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686728 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686900 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.686977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686931 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.687108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.686983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\") pod \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\" (UID: \"d7b62808-e524-42d2-b71b-01f7b2e51c2a\") " Oct 29 22:04:23.688459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.688428 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.689554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.689521 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.689727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.689595 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.690798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.690769 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "kube-api-access-ll8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:23.690900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.690841 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.691304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.691277 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:23.691872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.691843 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:23.692741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.692715 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.692839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.692732 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo" pod "d7b62808-e524-42d2-b71b-01f7b2e51c2a" (UID: "d7b62808-e524-42d2-b71b-01f7b2e51c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:23.789388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789351 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789387 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789401 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789415 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8j8\" (UniqueName: \"kubernetes.io/projected/d7b62808-e524-42d2-b71b-01f7b2e51c2a-kube-api-access-ll8j8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789429 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789443 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789456 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b62808-e524-42d2-b71b-01f7b2e51c2a-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789473 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.789611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.789488 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b62808-e524-42d2-b71b-01f7b2e51c2a-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:23.866944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.866589 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 22:04:23.879263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.879236 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:04:23.880810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.880774 4543 generic.go:356] "Generic (PLEG): container finished" podID="22af8b47-d881-4832-874f-12b90413feb0" containerID="3aefc17de225d12a960947c2cbf2b745f1136c49265fa3a3ae40ee76776f0868" exitCode=0 Oct 29 22:04:23.880936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.880886 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" event={"ID":"22af8b47-d881-4832-874f-12b90413feb0","Type":"ContainerDied","Data":"3aefc17de225d12a960947c2cbf2b745f1136c49265fa3a3ae40ee76776f0868"} Oct 29 22:04:23.880992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.880934 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" event={"ID":"22af8b47-d881-4832-874f-12b90413feb0","Type":"ContainerDied","Data":"ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e"} Oct 29 22:04:23.880992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.880966 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad2dce332bd251b5c76ead28f08da14401a6d246019ee29577124eb060e53f3e" Oct 29 22:04:23.882667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882644 4543 generic.go:356] "Generic (PLEG): container finished" podID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerID="0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf" exitCode=0 Oct 29 22:04:23.882781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882688 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" Oct 29 22:04:23.882781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882728 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" event={"ID":"d7b62808-e524-42d2-b71b-01f7b2e51c2a","Type":"ContainerDied","Data":"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf"} Oct 29 22:04:23.882781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882771 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx" event={"ID":"d7b62808-e524-42d2-b71b-01f7b2e51c2a","Type":"ContainerDied","Data":"483cda0bbd107f903f05659e34f28a8922b0609aa8f6d4b9ff9224fba0dbbaae"} Oct 29 22:04:23.882950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882801 4543 scope.go:122] "RemoveContainer" containerID="0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf" Oct 29 22:04:23.882950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.882846 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 22:04:23.886003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.885979 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" event={"ID":"ffe1ba54-13d2-4a4e-a0ef-53683cae0487","Type":"ContainerDied","Data":"79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a"} Oct 29 22:04:23.886141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.886025 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79907cd89199475a02836b493a1c7543ae97ceb7f7e85f75c05011a02984635a" Oct 29 22:04:23.886141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.886126 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd" Oct 29 22:04:23.893774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.893746 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.893870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.893848 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.893922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.893879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.893995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.893976 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894055 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894255 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894311 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894354 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894376 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\") pod \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\" (UID: \"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb\") " Oct 29 22:04:23.894867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.894848 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:04:23.895328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.895301 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.895432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.895316 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.895733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.895713 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.896260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.896235 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.896649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.896526 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:23.897328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.897302 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:23.899392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.899361 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:23.901465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.901223 4543 generic.go:356] "Generic (PLEG): container finished" podID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" containerID="d1e294183b83545350490da866d5bbef8ed1ed0f49f400073908527d1ce5570a" exitCode=0 Oct 29 22:04:23.901465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.901396 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" event={"ID":"5d39ff70-9797-4553-a1bd-03edf17a4fcd","Type":"ContainerDied","Data":"d1e294183b83545350490da866d5bbef8ed1ed0f49f400073908527d1ce5570a"} Oct 29 22:04:23.902930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.902848 4543 scope.go:122] "RemoveContainer" containerID="0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf" Oct 29 22:04:23.903497 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:23.903362 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf\": container with ID starting with 0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf not found: ID does not exist" containerID="0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf" Oct 29 22:04:23.903497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.903395 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf"} err="failed to get container status \"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf\": rpc error: code = NotFound desc = could not find container \"0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf\": container with ID starting with 0885d90c5c51c3175c26888bb8c8a147b9073dd00d2855f8818937b7ff7748cf not found: ID does not exist" Oct 29 22:04:23.903497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.903416 4543 scope.go:122] "RemoveContainer" containerID="d1e294183b83545350490da866d5bbef8ed1ed0f49f400073908527d1ce5570a" Oct 29 22:04:23.904612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.904301 4543 generic.go:356] "Generic (PLEG): container finished" podID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerID="68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7" exitCode=0 Oct 29 22:04:23.904612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.904368 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" event={"ID":"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb","Type":"ContainerDied","Data":"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7"} Oct 29 22:04:23.904612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.904397 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" event={"ID":"b3cf2d34-422f-4e62-83b5-c53ac1eabbfb","Type":"ContainerDied","Data":"42c4ae6c24d9dd95e1d9a23a99966910a191e8dd1f3fbc6c760a11ef55118eca"} Oct 29 22:04:23.904612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.904446 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl" Oct 29 22:04:23.906618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.906597 4543 generic.go:356] "Generic (PLEG): container finished" podID="00668c48-f9af-4b13-809e-68850c4c7642" containerID="2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577" exitCode=0 Oct 29 22:04:23.906721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.906638 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" event={"ID":"00668c48-f9af-4b13-809e-68850c4c7642","Type":"ContainerDied","Data":"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577"} Oct 29 22:04:23.906721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.906663 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" event={"ID":"00668c48-f9af-4b13-809e-68850c4c7642","Type":"ContainerDied","Data":"175a5fb666e237add257d90ed5b83e00d19b6d622f165e1f6b9d3feb32da9ca7"} Oct 29 22:04:23.906721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.906708 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p" Oct 29 22:04:23.916930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.916280 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:23.918480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.918437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs" pod "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" (UID: "b3cf2d34-422f-4e62-83b5-c53ac1eabbfb"). InnerVolumeSpecName "kube-api-access-z5njs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:23.920855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.920834 4543 scope.go:122] "RemoveContainer" containerID="68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7" Oct 29 22:04:23.965311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.962454 4543 scope.go:122] "RemoveContainer" containerID="68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7" Oct 29 22:04:23.974468 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:23.974439 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7\": container with ID starting with 68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7 not found: ID does not exist" containerID="68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7" Oct 29 22:04:23.974715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.974687 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7"} err="failed to get container status \"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7\": rpc error: code = NotFound desc = could not find container \"68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7\": container with ID starting with 68e48f0215d9a49a87c4a2cb7d54f68cacebcd3dfdeecd9472731cdc87dc25b7 not found: ID does not exist" Oct 29 22:04:23.974849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.974835 4543 scope.go:122] "RemoveContainer" containerID="2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577" Oct 29 22:04:23.991050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.991030 4543 scope.go:122] "RemoveContainer" containerID="2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577" Oct 29 22:04:23.991440 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:23.991419 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577\": container with ID starting with 2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577 not found: ID does not exist" containerID="2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577" Oct 29 22:04:23.991543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.991445 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577"} err="failed to get container status \"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577\": rpc error: code = NotFound desc = could not find container \"2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577\": container with ID starting with 2dece68e9c36b342895d2cad21177bffe3713b2ac063a95785b559489a794577 not found: ID does not exist" Oct 29 22:04:23.995535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995506 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995586 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\" (UniqueName: \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\") pod \"00668c48-f9af-4b13-809e-68850c4c7642\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " Oct 29 22:04:23.995637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995631 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995672 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995723 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995903 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.995996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.995980 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.996050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.996019 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.996205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.996052 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\") pod \"22af8b47-d881-4832-874f-12b90413feb0\" (UID: \"22af8b47-d881-4832-874f-12b90413feb0\") " Oct 29 22:04:23.996271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.996213 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\" (UniqueName: \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\") pod \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " Oct 29 22:04:23.996330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.996282 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\") pod \"00668c48-f9af-4b13-809e-68850c4c7642\" (UID: \"00668c48-f9af-4b13-809e-68850c4c7642\") " Oct 29 22:04:23.996386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.996350 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\") pod \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\" (UID: \"5d39ff70-9797-4553-a1bd-03edf17a4fcd\") " Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997258 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997284 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997301 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997315 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-kube-api-access-z5njs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997328 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997342 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997354 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997366 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997377 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997410 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.997769 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume" pod "5d39ff70-9797-4553-a1bd-03edf17a4fcd" (UID: "5d39ff70-9797-4553-a1bd-03edf17a4fcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.998292 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.998302 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.999502 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:23.999931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume" pod "00668c48-f9af-4b13-809e-68850c4c7642" (UID: "00668c48-f9af-4b13-809e-68850c4c7642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:24.005154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.000982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s" pod "00668c48-f9af-4b13-809e-68850c4c7642" (UID: "00668c48-f9af-4b13-809e-68850c4c7642"). InnerVolumeSpecName "kube-api-access-ct56s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.006498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.005583 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.017570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.017542 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.018015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.017989 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:24.018254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.018188 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "kube-api-access-5lv94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.018254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.018212 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc" pod "5d39ff70-9797-4553-a1bd-03edf17a4fcd" (UID: "5d39ff70-9797-4553-a1bd-03edf17a4fcd"). InnerVolumeSpecName "kube-api-access-84tmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:24.020602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.020581 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy" pod "22af8b47-d881-4832-874f-12b90413feb0" (UID: "22af8b47-d881-4832-874f-12b90413feb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:24.101403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.101373 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.101872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.101855 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.101996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.101981 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lv94\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-kube-api-access-5lv94\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.102106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.102093 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22af8b47-d881-4832-874f-12b90413feb0-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.102195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.102184 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tmc\" (UniqueName: \"kubernetes.io/projected/5d39ff70-9797-4553-a1bd-03edf17a4fcd-kube-api-access-84tmc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.102712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.102692 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00668c48-f9af-4b13-809e-68850c4c7642-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.102859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.102847 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d39ff70-9797-4553-a1bd-03edf17a4fcd-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.102947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.102935 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.103065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.103050 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct56s\" (UniqueName: \"kubernetes.io/projected/00668c48-f9af-4b13-809e-68850c4c7642-kube-api-access-ct56s\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.103183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.103167 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.103267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.103256 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22af8b47-d881-4832-874f-12b90413feb0-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.103349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.103339 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22af8b47-d881-4832-874f-12b90413feb0-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.103429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.103419 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22af8b47-d881-4832-874f-12b90413feb0-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:24.169209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.169157 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 22:04:24.176128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.176054 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-z2xrx"] Oct 29 22:04:24.275765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.275734 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 22:04:24.283029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.282998 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-kh7xl"] Oct 29 22:04:24.349159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.327307 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:04:24.371324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.371294 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-7987946448-jbz5p"] Oct 29 22:04:24.392715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.392681 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00668c48-f9af-4b13-809e-68850c4c7642" path="/var/lib/kubelet/pods/00668c48-f9af-4b13-809e-68850c4c7642/volumes" Oct 29 22:04:24.393275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.393260 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" path="/var/lib/kubelet/pods/b3cf2d34-422f-4e62-83b5-c53ac1eabbfb/volumes" Oct 29 22:04:24.393724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.393711 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" path="/var/lib/kubelet/pods/d7b62808-e524-42d2-b71b-01f7b2e51c2a/volumes" Oct 29 22:04:24.919653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.919628 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" Oct 29 22:04:24.919653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.919631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js" event={"ID":"5d39ff70-9797-4553-a1bd-03edf17a4fcd","Type":"ContainerDied","Data":"bbe9d3aa37f8d024bad32023b1e479b792cf80edf137d27a0a5b768b1ddc6b6a"} Oct 29 22:04:24.920819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.920796 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx" Oct 29 22:04:24.995287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:24.995256 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 22:04:25.008776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:25.008749 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-mscxx"] Oct 29 22:04:25.012907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:25.012874 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:04:25.056841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:25.056814 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:04:25.062327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:25.062305 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:04:25.066059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:25.066040 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad8e69-777c868775-nq4js"] Oct 29 22:04:26.083182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:26.080703 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt"] Oct 29 22:04:26.086337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:26.086307 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4722f60b-2hdqt"] Oct 29 22:04:26.387996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:26.387911 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22af8b47-d881-4832-874f-12b90413feb0" path="/var/lib/kubelet/pods/22af8b47-d881-4832-874f-12b90413feb0/volumes" Oct 29 22:04:26.388544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:26.388517 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" path="/var/lib/kubelet/pods/5d39ff70-9797-4553-a1bd-03edf17a4fcd/volumes" Oct 29 22:04:26.389022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:26.389003 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c86ac86-6cfe-423a-a6dd-f890de2fb48c" path="/var/lib/kubelet/pods/7c86ac86-6cfe-423a-a6dd-f890de2fb48c/volumes" Oct 29 22:04:27.696254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:27.696221 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:04:27.964009 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:27.963972 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:04:30.013174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.013130 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" podUID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" containerName="payload-processing" containerID="cri-o://12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9" gracePeriod=25 Oct 29 22:04:30.063211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.063161 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" podUID="8c87c450-426c-4ede-84a6-6b839f53c082" containerName="payload-pre-processing" containerID="cri-o://1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513" gracePeriod=25 Oct 29 22:04:30.358742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.358710 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:04:30.380490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.380469 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:04:30.534625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.534549 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\") pod \"8c87c450-426c-4ede-84a6-6b839f53c082\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " Oct 29 22:04:30.534625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.534592 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\" (UniqueName: \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\") pod \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " Oct 29 22:04:30.534864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.534664 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\") pod \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\" (UID: \"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c\") " Oct 29 22:04:30.534864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.534732 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\" (UniqueName: \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\") pod \"8c87c450-426c-4ede-84a6-6b839f53c082\" (UID: \"8c87c450-426c-4ede-84a6-6b839f53c082\") " Oct 29 22:04:30.535129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.535066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume" pod "8c87c450-426c-4ede-84a6-6b839f53c082" (UID: "8c87c450-426c-4ede-84a6-6b839f53c082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:30.535274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.535171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume" pod "f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" (UID: "f42eceb4-ce3f-4d75-83d8-53d2ae72b40c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:30.535949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.535925 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c87c450-426c-4ede-84a6-6b839f53c082-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:30.536044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.535955 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:30.537022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.537001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd" pod "f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" (UID: "f42eceb4-ce3f-4d75-83d8-53d2ae72b40c"). InnerVolumeSpecName "kube-api-access-pfssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:30.537151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.537135 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2" pod "8c87c450-426c-4ede-84a6-6b839f53c082" (UID: "8c87c450-426c-4ede-84a6-6b839f53c082"). InnerVolumeSpecName "kube-api-access-xs8x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:30.637393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.637366 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs8x2\" (UniqueName: \"kubernetes.io/projected/8c87c450-426c-4ede-84a6-6b839f53c082-kube-api-access-xs8x2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:30.637393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.637392 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfssd\" (UniqueName: \"kubernetes.io/projected/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c-kube-api-access-pfssd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:30.989159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.989123 4543 generic.go:356] "Generic (PLEG): container finished" podID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerID="73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa" exitCode=0 Oct 29 22:04:30.989337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.989196 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerDied","Data":"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa"} Oct 29 22:04:30.989663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.989644 4543 scope.go:122] "RemoveContainer" containerID="73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa" Oct 29 22:04:30.990783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.990758 4543 generic.go:356] "Generic (PLEG): container finished" podID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" containerID="12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9" exitCode=0 Oct 29 22:04:30.990874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.990845 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" Oct 29 22:04:30.990940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.990846 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" event={"ID":"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c","Type":"ContainerDied","Data":"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9"} Oct 29 22:04:30.991016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.990999 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s" event={"ID":"f42eceb4-ce3f-4d75-83d8-53d2ae72b40c","Type":"ContainerDied","Data":"29ee59d40fcb782cce9dd8f1d09a90d9ccb7a9b30a349ba2b0c4ee1cedcf2019"} Oct 29 22:04:30.991122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.991047 4543 scope.go:122] "RemoveContainer" containerID="12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9" Oct 29 22:04:30.991473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.991439 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:30.991578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.991535 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:30.991650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.991600 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:30.992533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.992511 4543 generic.go:356] "Generic (PLEG): container finished" podID="8c87c450-426c-4ede-84a6-6b839f53c082" containerID="1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513" exitCode=0 Oct 29 22:04:30.992623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.992555 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" Oct 29 22:04:30.992623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.992591 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" event={"ID":"8c87c450-426c-4ede-84a6-6b839f53c082","Type":"ContainerDied","Data":"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513"} Oct 29 22:04:30.992894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:30.992641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67" event={"ID":"8c87c450-426c-4ede-84a6-6b839f53c082","Type":"ContainerDied","Data":"95541d62d885c4d6b9e6273d4529f114c3715f5767c91736c8004538ec22b93b"} Oct 29 22:04:31.010310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.010287 4543 scope.go:122] "RemoveContainer" containerID="12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9" Oct 29 22:04:31.010663 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:31.010646 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9\": container with ID starting with 12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9 not found: ID does not exist" containerID="12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9" Oct 29 22:04:31.010714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.010671 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9"} err="failed to get container status \"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9\": rpc error: code = NotFound desc = could not find container \"12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9\": container with ID starting with 12ed5d11da2b2b1398be3c514fc5e58284fbafdb492e0d4deabf725822cdc0d9 not found: ID does not exist" Oct 29 22:04:31.010714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.010687 4543 scope.go:122] "RemoveContainer" containerID="1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513" Oct 29 22:04:31.023063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.023044 4543 scope.go:122] "RemoveContainer" containerID="1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513" Oct 29 22:04:31.023476 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:31.023456 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513\": container with ID starting with 1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513 not found: ID does not exist" containerID="1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513" Oct 29 22:04:31.023530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.023485 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513"} err="failed to get container status \"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513\": rpc error: code = NotFound desc = could not find container \"1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513\": container with ID starting with 1bdfbd82762acc09029e99ddd4e57245670bb319850f7fecf56b2e7a91e92513 not found: ID does not exist" Oct 29 22:04:31.088619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.088588 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:04:31.095607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.095418 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-442dc2-5b49cd7b95-smc67"] Oct 29 22:04:31.129920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.127489 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:04:31.134865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:31.134839 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-442dc2-58c8cd6854-hl69s"] Oct 29 22:04:32.010039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.010003 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerStarted","Data":"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73"} Oct 29 22:04:32.389092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.388992 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c87c450-426c-4ede-84a6-6b839f53c082" path="/var/lib/kubelet/pods/8c87c450-426c-4ede-84a6-6b839f53c082/volumes" Oct 29 22:04:32.389710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.389685 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" path="/var/lib/kubelet/pods/f42eceb4-ce3f-4d75-83d8-53d2ae72b40c/volumes" Oct 29 22:04:32.948095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.947651 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:04:32.948095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.947781 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:04:32.948095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:32.947841 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 29 22:04:33.953489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:33.953454 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:04:33.954024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:33.953988 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 29 22:04:34.952099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:34.945060 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:04:34.952099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:34.945152 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 29 22:04:35.949098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:35.948387 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:04:35.949098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:35.948459 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 29 22:04:36.058304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:36.058271 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs"] Oct 29 22:04:36.062719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:36.062695 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f49ac2-4fvbs"] Oct 29 22:04:36.386587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:36.386555 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b804c076-6c83-47d5-8fc1-0b259492dee4" path="/var/lib/kubelet/pods/b804c076-6c83-47d5-8fc1-0b259492dee4/volumes" Oct 29 22:04:36.949416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:36.949380 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:04:36.950001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:36.949969 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 29 22:04:37.949324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:37.949294 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:04:37.949720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:37.949698 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:04:37.950386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:37.950368 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:04:39.202920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:39.202893 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859"] Oct 29 22:04:39.215606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:39.215579 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fcce6b-h9859"] Oct 29 22:04:40.386980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:40.386948 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fadf590-bed5-4d97-b17a-d8e5c247b180" path="/var/lib/kubelet/pods/8fadf590-bed5-4d97-b17a-d8e5c247b180/volumes" Oct 29 22:04:42.718654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:42.718594 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" containerID="cri-o://0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e" gracePeriod=45 Oct 29 22:04:43.041987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.041963 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:04:43.143889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.143861 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\" (UniqueName: \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.143915 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.143966 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144181 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144265 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:43.144447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144286 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:43.144447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144294 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\") pod \"4bb32330-e422-4ade-9002-8a6f190c48a5\" (UID: \"4bb32330-e422-4ade-9002-8a6f190c48a5\") " Oct 29 22:04:43.144555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.144464 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:43.145233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.145208 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:43.145233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.145237 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:43.145417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.145250 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:43.147239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.147216 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:43.147341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.147232 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:43.147636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.147618 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb" pod "4bb32330-e422-4ade-9002-8a6f190c48a5" (UID: "4bb32330-e422-4ade-9002-8a6f190c48a5"). InnerVolumeSpecName "kube-api-access-lg9pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:43.169365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.169342 4543 generic.go:356] "Generic (PLEG): container finished" podID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerID="0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e" exitCode=0 Oct 29 22:04:43.169483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.169390 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" Oct 29 22:04:43.169483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.169422 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" event={"ID":"4bb32330-e422-4ade-9002-8a6f190c48a5","Type":"ContainerDied","Data":"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e"} Oct 29 22:04:43.169483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.169465 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n" event={"ID":"4bb32330-e422-4ade-9002-8a6f190c48a5","Type":"ContainerDied","Data":"cbb35abe48ed20c26ffdb40aaea624e711fcf325db2b62abdefbcf373834ecbf"} Oct 29 22:04:43.169483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.169482 4543 scope.go:122] "RemoveContainer" containerID="0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e" Oct 29 22:04:43.180604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.180587 4543 scope.go:122] "RemoveContainer" containerID="0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e" Oct 29 22:04:43.180869 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:43.180847 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e\": container with ID starting with 0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e not found: ID does not exist" containerID="0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e" Oct 29 22:04:43.180974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.180873 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e"} err="failed to get container status \"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e\": rpc error: code = NotFound desc = could not find container \"0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e\": container with ID starting with 0a855e96be4222355ba0246accebb572c37856abc2eeb548c239818b7d2af23e not found: ID does not exist" Oct 29 22:04:43.230573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.230547 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:04:43.235649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.235623 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f8867e7d/e2e-lifecycle-model-f8867e7d-kserve-54f8cd7597-4kx2n"] Oct 29 22:04:43.246670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.246646 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb32330-e422-4ade-9002-8a6f190c48a5-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:43.246754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.246673 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9pb\" (UniqueName: \"kubernetes.io/projected/4bb32330-e422-4ade-9002-8a6f190c48a5-kube-api-access-lg9pb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:43.246754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:43.246688 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb32330-e422-4ade-9002-8a6f190c48a5-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:44.395186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:44.395142 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" path="/var/lib/kubelet/pods/4bb32330-e422-4ade-9002-8a6f190c48a5/volumes" Oct 29 22:04:47.062526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.062491 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn"] Oct 29 22:04:47.069597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.069560 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64cfc2a-svhpn"] Oct 29 22:04:47.406198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.403500 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:04:47.679310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.679237 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8"] Oct 29 22:04:47.680208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680181 4543 cpu_manager.go:414] "containerMap: removing container" podUID="867a1364-f023-438f-93c9-d6ea4861032a" containerName="revoke-keys" Oct 29 22:04:47.680208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680203 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867a1364-f023-438f-93c9-d6ea4861032a" containerName="revoke-keys" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680225 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680231 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680246 4543 cpu_manager.go:414] "containerMap: removing container" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680251 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680266 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680271 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680280 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" containerName="payload-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680286 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" containerName="payload-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680297 4543 cpu_manager.go:414] "containerMap: removing container" podUID="00668c48-f9af-4b13-809e-68850c4c7642" containerName="payload-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680302 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00668c48-f9af-4b13-809e-68850c4c7642" containerName="payload-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680310 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8c87c450-426c-4ede-84a6-6b839f53c082" containerName="payload-pre-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680315 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c87c450-426c-4ede-84a6-6b839f53c082" containerName="payload-pre-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680324 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680329 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680337 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ffe1ba54-13d2-4a4e-a0ef-53683cae0487" containerName="revoke-keys" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680342 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe1ba54-13d2-4a4e-a0ef-53683cae0487" containerName="revoke-keys" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680353 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" containerName="payload-pre-processing" Oct 29 22:04:47.680364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680358 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" containerName="payload-pre-processing" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680453 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe1ba54-13d2-4a4e-a0ef-53683cae0487" containerName="revoke-keys" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680461 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d39ff70-9797-4553-a1bd-03edf17a4fcd" containerName="payload-pre-processing" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680468 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="867a1364-f023-438f-93c9-d6ea4861032a" containerName="revoke-keys" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680476 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b62808-e524-42d2-b71b-01f7b2e51c2a" containerName="istio-proxy" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680482 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="22af8b47-d881-4832-874f-12b90413feb0" containerName="istio-proxy" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680489 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cf2d34-422f-4e62-83b5-c53ac1eabbfb" containerName="istio-proxy" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680496 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42eceb4-ce3f-4d75-83d8-53d2ae72b40c" containerName="payload-processing" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680502 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="00668c48-f9af-4b13-809e-68850c4c7642" containerName="payload-processing" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680509 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c87c450-426c-4ede-84a6-6b839f53c082" containerName="payload-pre-processing" Oct 29 22:04:47.680894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.680515 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb32330-e422-4ade-9002-8a6f190c48a5" containerName="main" Oct 29 22:04:47.683804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.683788 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.701464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.701443 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:04:47.701635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.701459 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:04:47.701916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.701897 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8"] Oct 29 22:04:47.767412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.767344 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:04:47.767813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.767790 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:04:47.807841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.807804 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.808255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.808225 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.808493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.808474 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2z9\" (UniqueName: \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.909921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.909890 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.910331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.909990 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.910331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.910024 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2z9\" (UniqueName: \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.910735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.910714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.912487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.912468 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.963748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.963715 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2z9\" (UniqueName: \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\") pod \"maas-api-revoke-keys-e2e-api-b-296d19-sz5m8\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:47.999117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:47.999069 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:48.325519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.325489 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8"] Oct 29 22:04:48.326220 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:04:48.326187 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d7a5bc_4844_49f0_9ecb_0e65c064e5b7.slice/crio-5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2 WatchSource:0}: Error finding container 5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2: Status 404 returned error can't find the container with id 5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2 Oct 29 22:04:48.393785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.393747 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1768a709-706b-47a1-ab45-c1d5713cb4f4" path="/var/lib/kubelet/pods/1768a709-706b-47a1-ab45-c1d5713cb4f4/volumes" Oct 29 22:04:48.618576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.615784 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:04:48.622621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.622597 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.653902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.648329 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad8e69\"" Oct 29 22:04:48.653902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.649129 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad8e69-dockercfg-kkthl\"" Oct 29 22:04:48.661601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.656255 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:04:48.749436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.729213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.765475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.765437 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrmz\" (UniqueName: \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.868815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.868745 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.869065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.869044 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrmz\" (UniqueName: \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.869768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.869738 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:48.896191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:48.896161 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrmz\" (UniqueName: \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\") pod \"payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:49.085915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.085268 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:49.086860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.086834 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 22:04:49.088289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.087089 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" containerID="cri-o://fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656" gracePeriod=30 Oct 29 22:04:49.319588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.319551 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" event={"ID":"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7","Type":"ContainerStarted","Data":"b0b02a7acbe99ec81cf13819d078932077c8d11cdb55c04c8e0a6627bc42c004"} Oct 29 22:04:49.319780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.319614 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" event={"ID":"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7","Type":"ContainerStarted","Data":"5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2"} Oct 29 22:04:49.382042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.381950 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" podStartSLOduration=2.381931479 podStartE2EDuration="2.381931479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:49.380373352 +0000 UTC m=+1351.122304026" watchObservedRunningTime="2026-10-29 22:04:49.381931479 +0000 UTC m=+1351.123862152" Oct 29 22:04:49.474883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.474850 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:04:49.874886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.874841 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn"] Oct 29 22:04:49.879560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:49.879531 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76c786-dkcfn"] Oct 29 22:04:50.356903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:50.356864 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" event={"ID":"317e9673-f09b-4719-972a-db8ac68a234f","Type":"ContainerStarted","Data":"92ba1bc603162bf8ce10f9b47c96a6e5f1d1d5eeccc1cc4656d3118cf374d3a2"} Oct 29 22:04:50.362066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:50.362038 4543 generic.go:356] "Generic (PLEG): container finished" podID="78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" containerID="b0b02a7acbe99ec81cf13819d078932077c8d11cdb55c04c8e0a6627bc42c004" exitCode=0 Oct 29 22:04:50.362310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:50.362283 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" event={"ID":"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7","Type":"ContainerDied","Data":"b0b02a7acbe99ec81cf13819d078932077c8d11cdb55c04c8e0a6627bc42c004"} Oct 29 22:04:50.390637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:50.390094 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867a1364-f023-438f-93c9-d6ea4861032a" path="/var/lib/kubelet/pods/867a1364-f023-438f-93c9-d6ea4861032a/volumes" Oct 29 22:04:51.662032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.662004 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:51.810101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.798729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\") pod \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " Oct 29 22:04:51.810101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.798906 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\" (UniqueName: \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\") pod \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " Oct 29 22:04:51.810101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.799038 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\") pod \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\" (UID: \"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7\") " Oct 29 22:04:51.814797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.812315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca" pod "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" (UID: "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:51.828138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.820851 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca" pod "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" (UID: "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:51.828138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.823887 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9" pod "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" (UID: "78d7a5bc-4844-49f0-9ecb-0e65c064e5b7"). InnerVolumeSpecName "kube-api-access-kr2z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:51.901800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.900902 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:51.901800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.900933 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:51.901800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:51.900942 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2z9\" (UniqueName: \"kubernetes.io/projected/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7-kube-api-access-kr2z9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.389877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.389834 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" event={"ID":"317e9673-f09b-4719-972a-db8ac68a234f","Type":"ContainerStarted","Data":"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26"} Oct 29 22:04:52.390111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.389896 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:04:52.390111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.389921 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" event={"ID":"78d7a5bc-4844-49f0-9ecb-0e65c064e5b7","Type":"ContainerDied","Data":"5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2"} Oct 29 22:04:52.390111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.389945 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d51403074fcfe5faa804a1bca882f0fd32a3be9591678e004c0bbfa0cd78ed2" Oct 29 22:04:52.390111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.389966 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8" Oct 29 22:04:52.404024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.403975 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" podUID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" containerName="payload-pre-processing" containerID="cri-o://5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d" gracePeriod=25 Oct 29 22:04:52.440354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.440304 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" podStartSLOduration=2.3981584959999998 podStartE2EDuration="4.440289813s" totalImagesPullingTime="2.042131317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:52.433313343 +0000 UTC m=+1354.175244016" watchObservedRunningTime="2026-10-29 22:04:52.440289813 +0000 UTC m=+1354.182220526" Oct 29 22:04:52.745409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.745380 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:04:52.923243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.923211 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\") pod \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " Oct 29 22:04:52.923467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.923443 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\" (UniqueName: \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\") pod \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\" (UID: \"d69af68f-63ef-4740-b446-f2fc3c19f7b6\") " Oct 29 22:04:52.923724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.923697 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume" pod "d69af68f-63ef-4740-b446-f2fc3c19f7b6" (UID: "d69af68f-63ef-4740-b446-f2fc3c19f7b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:52.924254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.924186 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69af68f-63ef-4740-b446-f2fc3c19f7b6-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.926020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:52.925996 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc" pod "d69af68f-63ef-4740-b446-f2fc3c19f7b6" (UID: "d69af68f-63ef-4740-b446-f2fc3c19f7b6"). InnerVolumeSpecName "kube-api-access-zkjbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:53.026227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.026188 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjbc\" (UniqueName: \"kubernetes.io/projected/d69af68f-63ef-4740-b446-f2fc3c19f7b6-kube-api-access-zkjbc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:53.420972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.420814 4543 generic.go:356] "Generic (PLEG): container finished" podID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" containerID="5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d" exitCode=0 Oct 29 22:04:53.426258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.426229 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" Oct 29 22:04:53.428759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.426475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" event={"ID":"d69af68f-63ef-4740-b446-f2fc3c19f7b6","Type":"ContainerDied","Data":"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d"} Oct 29 22:04:53.429339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.429144 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8" event={"ID":"d69af68f-63ef-4740-b446-f2fc3c19f7b6","Type":"ContainerDied","Data":"6dbb54bc56375b67446039811e24e1e177c4cada8bfadaed0ff12f6dfcce2950"} Oct 29 22:04:53.429525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.429508 4543 scope.go:122] "RemoveContainer" containerID="5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d" Oct 29 22:04:53.460193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.460160 4543 scope.go:122] "RemoveContainer" containerID="5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d" Oct 29 22:04:53.460607 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:53.460583 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d\": container with ID starting with 5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d not found: ID does not exist" containerID="5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d" Oct 29 22:04:53.460710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.460618 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d"} err="failed to get container status \"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d\": rpc error: code = NotFound desc = could not find container \"5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d\": container with ID starting with 5b042d86ac11b8bc5b24f0ffd27167509615436fde4f5fcf51d66a45a339cf8d not found: ID does not exist" Oct 29 22:04:53.490760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.490735 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:04:53.504222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:53.504199 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fe45b-565b7dc5c5-vb8r8"] Oct 29 22:04:54.379784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.379730 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 22:04:54.391723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.391685 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" path="/var/lib/kubelet/pods/d69af68f-63ef-4740-b446-f2fc3c19f7b6/volumes" Oct 29 22:04:54.431951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.431923 4543 generic.go:356] "Generic (PLEG): container finished" podID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerID="fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656" exitCode=0 Oct 29 22:04:54.432068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.431981 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" Oct 29 22:04:54.432068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.432093 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" event={"ID":"f7a3bfb9-9b82-47c7-8092-525e92bfca3f","Type":"ContainerDied","Data":"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656"} Oct 29 22:04:54.432215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.432127 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6" event={"ID":"f7a3bfb9-9b82-47c7-8092-525e92bfca3f","Type":"ContainerDied","Data":"efd8e197760deec6d539942eaf0884bb335fca934e3bd49370f4ac0d6bf00d7a"} Oct 29 22:04:54.432215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.432147 4543 scope.go:122] "RemoveContainer" containerID="fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656" Oct 29 22:04:54.443623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.443434 4543 scope.go:122] "RemoveContainer" containerID="fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656" Oct 29 22:04:54.443774 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:54.443748 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656\": container with ID starting with fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656 not found: ID does not exist" containerID="fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656" Oct 29 22:04:54.443837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.443786 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656"} err="failed to get container status \"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656\": rpc error: code = NotFound desc = could not find container \"fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656\": container with ID starting with fdd61522d050a4d064bfad233158fd727dccd3355da49f36bfb3f33b5d3f3656 not found: ID does not exist" Oct 29 22:04:54.556937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.556866 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.557193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.556956 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.557193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557010 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.557721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557686 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557734 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557856 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557922 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557948 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558006 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\") pod \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\" (UID: \"f7a3bfb9-9b82-47c7-8092-525e92bfca3f\") " Oct 29 22:04:54.558374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.557684 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:54.558698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558673 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558915 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558938 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.558345 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.559229 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:54.561416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.561395 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:54.562662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.562608 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:54.562991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.562965 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:54.563270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.563246 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g" pod "f7a3bfb9-9b82-47c7-8092-525e92bfca3f" (UID: "f7a3bfb9-9b82-47c7-8092-525e92bfca3f"). InnerVolumeSpecName "kube-api-access-xj25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660029 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660064 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660088 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660097 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660107 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660120 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj25g\" (UniqueName: \"kubernetes.io/projected/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-kube-api-access-xj25g\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.660421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.660136 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a3bfb9-9b82-47c7-8092-525e92bfca3f-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:04:54.792137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.792099 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 22:04:54.805001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:54.804964 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-hkmq6"] Oct 29 22:04:55.444103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.444013 4543 generic.go:356] "Generic (PLEG): container finished" podID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerID="6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73" exitCode=0 Oct 29 22:04:55.444483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.444107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerDied","Data":"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73"} Oct 29 22:04:55.444581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.444559 4543 scope.go:122] "RemoveContainer" containerID="6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73" Oct 29 22:04:55.446319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.446284 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:55.446404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.446363 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:55.446460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:55.446404 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:56.387647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:56.387613 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" path="/var/lib/kubelet/pods/f7a3bfb9-9b82-47c7-8092-525e92bfca3f/volumes" Oct 29 22:04:56.608248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:56.586952 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerStarted","Data":"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0"} Oct 29 22:04:56.972100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:56.972045 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd"] Oct 29 22:04:56.980093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:56.980047 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fd295-7sccd"] Oct 29 22:04:57.226236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.226134 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:04:57.233745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.233712 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:04:57.234014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.233984 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:04:57.960444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.960409 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.975278 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976229 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976250 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976277 4543 cpu_manager.go:414] "containerMap: removing container" podUID="78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" containerName="revoke-keys" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976284 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" containerName="revoke-keys" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976335 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" containerName="payload-pre-processing" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976344 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" containerName="payload-pre-processing" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976501 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69af68f-63ef-4740-b446-f2fc3c19f7b6" containerName="payload-pre-processing" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976525 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a3bfb9-9b82-47c7-8092-525e92bfca3f" containerName="istio-proxy" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.976540 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" containerName="revoke-keys" Oct 29 22:04:57.980094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:57.979965 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:57.995128 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:04:57.989669 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:04:58.010017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.009995 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:04:58.015149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.015128 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7046db-openshift-default-dockercfg-mxqkx\"" Oct 29 22:04:58.026509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026482 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026525 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscth\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026677 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026719 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026764 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.026908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.026889 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.027035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.027005 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.027139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.027119 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135288 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135572 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135599 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135632 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscth\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135696 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135726 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135782 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.135840 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.136362 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.146102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.136611 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.180096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.168691 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.180096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.169608 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.180096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.169917 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.227101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.226900 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:04:58.227101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.226956 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:04:58.273109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.252742 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.357288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.357253 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:04:58.358280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.358234 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" containerID="cri-o://b2c494824d5e32eeebb116ed0da3bcff505c96cdc635a4da72fa61ed2ae493db" gracePeriod=30 Oct 29 22:04:58.400173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.399345 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscth\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.486932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.486835 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.491709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.491675 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\") pod \"e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:58.497029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.497001 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe1ba54-13d2-4a4e-a0ef-53683cae0487" path="/var/lib/kubelet/pods/ffe1ba54-13d2-4a4e-a0ef-53683cae0487/volumes" Oct 29 22:04:58.613806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:58.596571 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:04:59.074293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.059495 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8"] Oct 29 22:04:59.086690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.086653 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:04:59.088116 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:04:59.088061 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dce852_8447_4004_989d_86ee0f6433a3.slice/crio-c820b16a2e346a1cf1281e9bc3ae2b4081d91a1235d9269e8d0b036016f39ad1 WatchSource:0}: Error finding container c820b16a2e346a1cf1281e9bc3ae2b4081d91a1235d9269e8d0b036016f39ad1: Status 404 returned error can't find the container with id c820b16a2e346a1cf1281e9bc3ae2b4081d91a1235d9269e8d0b036016f39ad1 Oct 29 22:04:59.096574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.092933 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:59.096574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.093028 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:59.096574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.093091 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:04:59.096574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.093720 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q"] Oct 29 22:04:59.122402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.119324 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40db6d2f-924j8"] Oct 29 22:04:59.122402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.119386 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q"] Oct 29 22:04:59.122402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.119554 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.124258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.123673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.124258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.123757 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tfz\" (UniqueName: \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.124258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.123947 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.141740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.141713 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:04:59.142031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.142013 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:04:59.142788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.142768 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:04:59.142976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.142960 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:04:59.218166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.218135 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:04:59.218344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.218189 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:04:59.225141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.225106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.225289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.225276 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.225335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.225325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tfz\" (UniqueName: \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.225706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.225684 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.227945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.227920 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.259099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.257615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tfz\" (UniqueName: \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.516112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.513357 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:04:59.695120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.694445 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerStarted","Data":"e75f2de3effe67c49540147ef2d166ddae21ebe2385e749e8327198ba4e33a04"} Oct 29 22:04:59.695120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.694511 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerStarted","Data":"c820b16a2e346a1cf1281e9bc3ae2b4081d91a1235d9269e8d0b036016f39ad1"} Oct 29 22:04:59.774105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.773030 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podStartSLOduration=2.773009558 podStartE2EDuration="2.773009558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:59.769663089 +0000 UTC m=+1361.511593762" watchObservedRunningTime="2026-10-29 22:04:59.773009558 +0000 UTC m=+1361.514940233" Oct 29 22:04:59.991506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:04:59.991256 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q"] Oct 29 22:05:00.217284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.217251 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:05:00.217796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.217309 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:05:00.387952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.387915 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8b12e0-9103-4f6e-9d1f-70800c12a353" path="/var/lib/kubelet/pods/5a8b12e0-9103-4f6e-9d1f-70800c12a353/volumes" Oct 29 22:05:00.600556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.600477 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:05:00.606265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.606243 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:05:00.705455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.705411 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" event={"ID":"d5f3892d-ad84-4390-b4e7-584f6387a591","Type":"ContainerStarted","Data":"43b6518c401112baed1eb983116ddab0c1b3bd1cdd2b57be8aaecd28b54087d7"} Oct 29 22:05:00.705617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.705471 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" event={"ID":"d5f3892d-ad84-4390-b4e7-584f6387a591","Type":"ContainerStarted","Data":"48e639c546adf37c0abf0980a81404adce133be20bf1ce184e20e5eef0400029"} Oct 29 22:05:00.705617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.705541 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:05:00.706463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.706447 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:05:00.757982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:00.757923 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" podStartSLOduration=1.757911655 podStartE2EDuration="1.757911655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:00.753459684 +0000 UTC m=+1362.495390357" watchObservedRunningTime="2026-10-29 22:05:00.757911655 +0000 UTC m=+1362.499842327" Oct 29 22:05:01.217499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:01.217466 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:05:01.217862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:01.217531 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:05:01.716869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:01.716722 4543 generic.go:356] "Generic (PLEG): container finished" podID="d5f3892d-ad84-4390-b4e7-584f6387a591" containerID="43b6518c401112baed1eb983116ddab0c1b3bd1cdd2b57be8aaecd28b54087d7" exitCode=0 Oct 29 22:05:01.716869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:01.716762 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" event={"ID":"d5f3892d-ad84-4390-b4e7-584f6387a591","Type":"ContainerDied","Data":"43b6518c401112baed1eb983116ddab0c1b3bd1cdd2b57be8aaecd28b54087d7"} Oct 29 22:05:02.218286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:02.218256 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:05:02.218663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:02.218334 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 29 22:05:02.917020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:02.916992 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:05:03.040014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.039942 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\" (UniqueName: \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\") pod \"d5f3892d-ad84-4390-b4e7-584f6387a591\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " Oct 29 22:05:03.040223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.040020 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\") pod \"d5f3892d-ad84-4390-b4e7-584f6387a591\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " Oct 29 22:05:03.040223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.040147 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\") pod \"d5f3892d-ad84-4390-b4e7-584f6387a591\" (UID: \"d5f3892d-ad84-4390-b4e7-584f6387a591\") " Oct 29 22:05:03.040504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.040480 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca" pod "d5f3892d-ad84-4390-b4e7-584f6387a591" (UID: "d5f3892d-ad84-4390-b4e7-584f6387a591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:03.040854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.040835 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:03.042275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.042255 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca" pod "d5f3892d-ad84-4390-b4e7-584f6387a591" (UID: "d5f3892d-ad84-4390-b4e7-584f6387a591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:03.042399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.042386 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz" pod "d5f3892d-ad84-4390-b4e7-584f6387a591" (UID: "d5f3892d-ad84-4390-b4e7-584f6387a591"). InnerVolumeSpecName "kube-api-access-v5tfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:03.141944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.141920 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5tfz\" (UniqueName: \"kubernetes.io/projected/d5f3892d-ad84-4390-b4e7-584f6387a591-kube-api-access-v5tfz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:03.141944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.141945 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f3892d-ad84-4390-b4e7-584f6387a591-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:03.222284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.222261 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:05:03.223355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.223338 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:05:03.437917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.437890 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:05:03.737609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.737583 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" Oct 29 22:05:03.737786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.737619 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q" event={"ID":"d5f3892d-ad84-4390-b4e7-584f6387a591","Type":"ContainerDied","Data":"48e639c546adf37c0abf0980a81404adce133be20bf1ce184e20e5eef0400029"} Oct 29 22:05:03.737786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.737670 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e639c546adf37c0abf0980a81404adce133be20bf1ce184e20e5eef0400029" Oct 29 22:05:03.739794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.739766 4543 generic.go:356] "Generic (PLEG): container finished" podID="81029b9d-4392-4638-ad49-600ac5dd1996" containerID="b2c494824d5e32eeebb116ed0da3bcff505c96cdc635a4da72fa61ed2ae493db" exitCode=0 Oct 29 22:05:03.739916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.739830 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerDied","Data":"b2c494824d5e32eeebb116ed0da3bcff505c96cdc635a4da72fa61ed2ae493db"} Oct 29 22:05:03.739916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:03.739882 4543 scope.go:122] "RemoveContainer" containerID="2618bc8622c7a9a403dcdfe914d49ef2da301be712628f8d8e3d3b3445ce1bec" Oct 29 22:05:04.103206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.103179 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:05:04.157536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157507 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157578 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157616 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157776 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157884 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157925 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.157965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.157931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:04.158065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:04.158135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158061 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.158135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158127 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.158243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158166 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\") pod \"81029b9d-4392-4638-ad49-600ac5dd1996\" (UID: \"81029b9d-4392-4638-ad49-600ac5dd1996\") " Oct 29 22:05:04.158346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158302 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:04.158346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158307 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:04.158687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158664 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:04.158975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158947 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.158975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158976 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81029b9d-4392-4638-ad49-600ac5dd1996-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.159202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.158989 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.159202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.159003 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.159202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.159015 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.160211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.160178 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:04.160901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.160868 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:04.197551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.197517 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:04.199278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.199248 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng" pod "81029b9d-4392-4638-ad49-600ac5dd1996" (UID: "81029b9d-4392-4638-ad49-600ac5dd1996"). InnerVolumeSpecName "kube-api-access-g9qng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.204774 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.205871 4543 cpu_manager.go:414] "containerMap: removing container" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.205891 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.205907 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d5f3892d-ad84-4390-b4e7-584f6387a591" containerName="revoke-keys" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.205917 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f3892d-ad84-4390-b4e7-584f6387a591" containerName="revoke-keys" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.206149 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.206175 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:04.207133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.206187 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f3892d-ad84-4390-b4e7-584f6387a591" containerName="revoke-keys" Oct 29 22:05:04.209637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.209565 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.212246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.212223 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:05:04.227916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.227894 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7046db-dockercfg-f5bp7\"" Oct 29 22:05:04.228319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.228002 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7046db\"" Oct 29 22:05:04.260720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260674 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.260720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260717 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7z5\" (UniqueName: \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.261121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260926 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.261121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260940 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9qng\" (UniqueName: \"kubernetes.io/projected/81029b9d-4392-4638-ad49-600ac5dd1996-kube-api-access-g9qng\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.261121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260958 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81029b9d-4392-4638-ad49-600ac5dd1996-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.261121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.260966 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81029b9d-4392-4638-ad49-600ac5dd1996-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:04.373038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.371344 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.373038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.371394 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7z5\" (UniqueName: \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.373038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.372929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.398621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.398591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7z5\" (UniqueName: \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\") pod \"payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.530545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.530511 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:04.749738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.749705 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" Oct 29 22:05:04.749912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.749741 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh" event={"ID":"81029b9d-4392-4638-ad49-600ac5dd1996","Type":"ContainerDied","Data":"1d279201bc127be86868db7590440c5793ff255c1e2474c58a2af299b3324a76"} Oct 29 22:05:04.749912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.749804 4543 scope.go:122] "RemoveContainer" containerID="b2c494824d5e32eeebb116ed0da3bcff505c96cdc635a4da72fa61ed2ae493db" Oct 29 22:05:04.811437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.811412 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:05:04.824705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.824680 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-76c786-openshift-default-5d9785fb65-xw4sh"] Oct 29 22:05:04.858164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:04.858133 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:05:04.858416 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:04.858395 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1fba0d_78f6_4d4a_b0c5_57011b3b7c2d.slice/crio-be6d98ecf7d9d8ba2e7d197de6d4db62fdfc899eb0ae54f8c8936f59cee142b4 WatchSource:0}: Error finding container be6d98ecf7d9d8ba2e7d197de6d4db62fdfc899eb0ae54f8c8936f59cee142b4: Status 404 returned error can't find the container with id be6d98ecf7d9d8ba2e7d197de6d4db62fdfc899eb0ae54f8c8936f59cee142b4 Oct 29 22:05:05.759898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:05.759861 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" event={"ID":"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d","Type":"ContainerStarted","Data":"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4"} Oct 29 22:05:05.760388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:05.759953 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:05.760388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:05.759976 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" event={"ID":"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d","Type":"ContainerStarted","Data":"be6d98ecf7d9d8ba2e7d197de6d4db62fdfc899eb0ae54f8c8936f59cee142b4"} Oct 29 22:05:05.805842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:05.805796 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" podStartSLOduration=1.805782703 podStartE2EDuration="1.805782703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:05.804149937 +0000 UTC m=+1367.546080624" watchObservedRunningTime="2026-10-29 22:05:05.805782703 +0000 UTC m=+1367.547713375" Oct 29 22:05:06.044877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.044799 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 22:05:06.045062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.045039 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" containerID="cri-o://c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a" gracePeriod=30 Oct 29 22:05:06.048820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.048796 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 22:05:06.049040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.049015 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" containerID="cri-o://da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca" gracePeriod=30 Oct 29 22:05:06.053517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.053492 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 22:05:06.053710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.053666 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" containerID="cri-o://d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f" gracePeriod=30 Oct 29 22:05:06.393187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.391724 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" path="/var/lib/kubelet/pods/81029b9d-4392-4638-ad49-600ac5dd1996/volumes" Oct 29 22:05:06.931690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.931662 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6"] Oct 29 22:05:06.932390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.932376 4543 cpu_manager.go:414] "containerMap: removing container" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:06.932390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.932390 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81029b9d-4392-4638-ad49-600ac5dd1996" containerName="istio-proxy" Oct 29 22:05:06.934931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.934914 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:06.956356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.956333 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:05:06.958594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.958044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:06.958594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.958151 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6njx\" (UniqueName: \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:06.958594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.958355 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:06.962776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.962747 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6"] Oct 29 22:05:06.969502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:06.969483 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:05:07.012711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.012582 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:05:07.013724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.013047 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:05:07.059826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.059795 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6njx\" (UniqueName: \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.060022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.060006 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.060178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.060160 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.060644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.060625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.062752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.062728 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.088371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.088347 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6njx\" (UniqueName: \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.249287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.249261 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:07.566860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.566834 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6"] Oct 29 22:05:07.567581 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:07.567548 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0d6af7_136d_4ce4_8777_ff93f1351825.slice/crio-ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640 WatchSource:0}: Error finding container ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640: Status 404 returned error can't find the container with id ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640 Oct 29 22:05:07.887348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.887318 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" event={"ID":"9b0d6af7-136d-4ce4-8777-ff93f1351825","Type":"ContainerStarted","Data":"5bd8a2cc316faeca4fbbb55f07492a9417251e5da3ea8af2993ecdfe736cb002"} Oct 29 22:05:07.887469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.887370 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" event={"ID":"9b0d6af7-136d-4ce4-8777-ff93f1351825","Type":"ContainerStarted","Data":"ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640"} Oct 29 22:05:07.932311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:07.932244 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" podStartSLOduration=1.932226561 podStartE2EDuration="1.932226561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:07.930744186 +0000 UTC m=+1369.672674872" watchObservedRunningTime="2026-10-29 22:05:07.932226561 +0000 UTC m=+1369.674157235" Oct 29 22:05:08.897295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:08.897257 4543 generic.go:356] "Generic (PLEG): container finished" podID="9b0d6af7-136d-4ce4-8777-ff93f1351825" containerID="5bd8a2cc316faeca4fbbb55f07492a9417251e5da3ea8af2993ecdfe736cb002" exitCode=0 Oct 29 22:05:08.897453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:08.897347 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" event={"ID":"9b0d6af7-136d-4ce4-8777-ff93f1351825","Type":"ContainerDied","Data":"5bd8a2cc316faeca4fbbb55f07492a9417251e5da3ea8af2993ecdfe736cb002"} Oct 29 22:05:10.087150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.087129 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:10.104004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.103971 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\") pod \"9b0d6af7-136d-4ce4-8777-ff93f1351825\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " Oct 29 22:05:10.104149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.104067 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\" (UniqueName: \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\") pod \"9b0d6af7-136d-4ce4-8777-ff93f1351825\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " Oct 29 22:05:10.104238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.104224 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\") pod \"9b0d6af7-136d-4ce4-8777-ff93f1351825\" (UID: \"9b0d6af7-136d-4ce4-8777-ff93f1351825\") " Oct 29 22:05:10.104531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.104504 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca" pod "9b0d6af7-136d-4ce4-8777-ff93f1351825" (UID: "9b0d6af7-136d-4ce4-8777-ff93f1351825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:10.105279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.105256 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:10.106871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.106849 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx" pod "9b0d6af7-136d-4ce4-8777-ff93f1351825" (UID: "9b0d6af7-136d-4ce4-8777-ff93f1351825"). InnerVolumeSpecName "kube-api-access-x6njx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:10.107033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.107014 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca" pod "9b0d6af7-136d-4ce4-8777-ff93f1351825" (UID: "9b0d6af7-136d-4ce4-8777-ff93f1351825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:10.206491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.206420 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0d6af7-136d-4ce4-8777-ff93f1351825-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:10.206491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.206451 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6njx\" (UniqueName: \"kubernetes.io/projected/9b0d6af7-136d-4ce4-8777-ff93f1351825-kube-api-access-x6njx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:10.915134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.915108 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" Oct 29 22:05:10.915303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.915099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6" event={"ID":"9b0d6af7-136d-4ce4-8777-ff93f1351825","Type":"ContainerDied","Data":"ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640"} Oct 29 22:05:10.915303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:10.915227 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6afe799699f0df390d22e55bc5fca189e3e7462fb4bbf20e39a32f744ad640" Oct 29 22:05:11.343992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.343935 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 22:05:11.376882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.376861 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 22:05:11.421712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421688 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421725 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421783 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421801 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421852 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421888 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.421924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421918 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.422336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421965 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.422336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.421969 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.422336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422214 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.422336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422274 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.422336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422322 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422273 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422421 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422385 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422402 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422480 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.422613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422624 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422701 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\") pod \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\" (UID: \"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6\") " Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422744 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422745 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422782 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\") pod \"7b6111b3-741b-43c8-9bee-4368f3817d0d\" (UID: \"7b6111b3-741b-43c8-9bee-4368f3817d0d\") " Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.422961 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423037 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:11.423651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423306 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.423651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423617 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.423651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423638 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6111b3-741b-43c8-9bee-4368f3817d0d-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.423651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423652 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423672 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423691 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423704 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423717 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423740 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.423753 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.424397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.424156 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.425453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.425430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:11.425599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.425582 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.425923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.425893 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.426610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.426574 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.427016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.426981 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.427243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.427223 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "kube-api-access-86j5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.427620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.427605 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo" pod "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" (UID: "3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:11.427727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.427709 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq" pod "7b6111b3-741b-43c8-9bee-4368f3817d0d" (UID: "7b6111b3-741b-43c8-9bee-4368f3817d0d"). InnerVolumeSpecName "kube-api-access-5vbbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.524768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524745 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.524768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524768 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524777 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524785 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524794 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86j5f\" (UniqueName: \"kubernetes.io/projected/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-kube-api-access-86j5f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524802 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6111b3-741b-43c8-9bee-4368f3817d0d-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524810 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524820 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbbq\" (UniqueName: \"kubernetes.io/projected/7b6111b3-741b-43c8-9bee-4368f3817d0d-kube-api-access-5vbbq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.525000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.524828 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.704021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.703996 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 22:05:11.828755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.828728 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.828908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.828798 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.828908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.828887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.828972 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829014 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829041 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829044 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.829173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829136 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.829337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829235 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.829337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829250 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829297 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.829468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829338 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829366 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\") pod \"39396390-b94e-42fa-ae91-e837573e4009\" (UID: \"39396390-b94e-42fa-ae91-e837573e4009\") " Oct 29 22:05:11.829559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.829519 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:11.830282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.830256 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.830405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.830288 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.830405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.830305 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.830405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.830318 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39396390-b94e-42fa-ae91-e837573e4009-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.830405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.830331 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.831430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.831404 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.831530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.831478 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:11.831530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.831502 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:11.831663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.831581 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r" pod "39396390-b94e-42fa-ae91-e837573e4009" (UID: "39396390-b94e-42fa-ae91-e837573e4009"). InnerVolumeSpecName "kube-api-access-hpc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:11.926638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.926552 4543 generic.go:356] "Generic (PLEG): container finished" podID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerID="d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f" exitCode=0 Oct 29 22:05:11.926638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.926607 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" Oct 29 22:05:11.926847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.926630 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" event={"ID":"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6","Type":"ContainerDied","Data":"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f"} Oct 29 22:05:11.926847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.926687 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn" event={"ID":"3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6","Type":"ContainerDied","Data":"5a7fdc3632089f563cc4cbb704f7fe0d63912f11dbc8883be93f6d08f80818da"} Oct 29 22:05:11.926847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.926700 4543 scope.go:122] "RemoveContainer" containerID="d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f" Oct 29 22:05:11.928349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.928325 4543 generic.go:356] "Generic (PLEG): container finished" podID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerID="c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a" exitCode=0 Oct 29 22:05:11.928473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.928373 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" Oct 29 22:05:11.928473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.928420 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" event={"ID":"7b6111b3-741b-43c8-9bee-4368f3817d0d","Type":"ContainerDied","Data":"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a"} Oct 29 22:05:11.928473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.928461 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s" event={"ID":"7b6111b3-741b-43c8-9bee-4368f3817d0d","Type":"ContainerDied","Data":"39da6e9a1938c252b1e08ecf022bf1ba9961c144535fbd7c349a656332c257eb"} Oct 29 22:05:11.930247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.930228 4543 generic.go:356] "Generic (PLEG): container finished" podID="39396390-b94e-42fa-ae91-e837573e4009" containerID="da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca" exitCode=0 Oct 29 22:05:11.930393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.930284 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" Oct 29 22:05:11.930393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.930310 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" event={"ID":"39396390-b94e-42fa-ae91-e837573e4009","Type":"ContainerDied","Data":"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca"} Oct 29 22:05:11.930393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.930364 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4" event={"ID":"39396390-b94e-42fa-ae91-e837573e4009","Type":"ContainerDied","Data":"d08e886431a0dbfb5fdbbf17f42c06dd789456bc4b8bd9ed550e4ed06987156f"} Oct 29 22:05:11.931243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.931220 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39396390-b94e-42fa-ae91-e837573e4009-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.931328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.931244 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpc9r\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-kube-api-access-hpc9r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.931328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.931255 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39396390-b94e-42fa-ae91-e837573e4009-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.931328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.931264 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39396390-b94e-42fa-ae91-e837573e4009-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:11.940501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.940481 4543 scope.go:122] "RemoveContainer" containerID="d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f" Oct 29 22:05:11.940808 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:11.940783 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f\": container with ID starting with d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f not found: ID does not exist" containerID="d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f" Oct 29 22:05:11.940911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.940819 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f"} err="failed to get container status \"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f\": rpc error: code = NotFound desc = could not find container \"d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f\": container with ID starting with d435cb2b40ce12469136e620611765aaf7b54b3e1726833e3a19a65870edef8f not found: ID does not exist" Oct 29 22:05:11.940911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.940841 4543 scope.go:122] "RemoveContainer" containerID="c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a" Oct 29 22:05:11.951172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.951154 4543 scope.go:122] "RemoveContainer" containerID="c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a" Oct 29 22:05:11.951428 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:11.951409 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a\": container with ID starting with c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a not found: ID does not exist" containerID="c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a" Oct 29 22:05:11.951484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.951434 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a"} err="failed to get container status \"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a\": rpc error: code = NotFound desc = could not find container \"c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a\": container with ID starting with c97ba538f4d5dcde1384191bf8436600c5d38348e45b60374a221a44e765fc3a not found: ID does not exist" Oct 29 22:05:11.951484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.951452 4543 scope.go:122] "RemoveContainer" containerID="da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca" Oct 29 22:05:11.961289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.961271 4543 scope.go:122] "RemoveContainer" containerID="da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca" Oct 29 22:05:11.961662 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:11.961637 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca\": container with ID starting with da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca not found: ID does not exist" containerID="da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca" Oct 29 22:05:11.961735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.961673 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca"} err="failed to get container status \"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca\": rpc error: code = NotFound desc = could not find container \"da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca\": container with ID starting with da4bbb217fc0cf560deb1549a3081b7ea955bc6df2ff0f463695aced8e27e2ca not found: ID does not exist" Oct 29 22:05:11.988460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.988437 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 22:05:11.993117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:11.993098 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-b5c2s"] Oct 29 22:05:12.032099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.032063 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 22:05:12.037283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.037265 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-lslbn"] Oct 29 22:05:12.080190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.080165 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 22:05:12.083946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.083925 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-8s9k4"] Oct 29 22:05:12.387439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.387408 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39396390-b94e-42fa-ae91-e837573e4009" path="/var/lib/kubelet/pods/39396390-b94e-42fa-ae91-e837573e4009/volumes" Oct 29 22:05:12.387895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.387881 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" path="/var/lib/kubelet/pods/3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6/volumes" Oct 29 22:05:12.388349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:12.388335 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" path="/var/lib/kubelet/pods/7b6111b3-741b-43c8-9bee-4368f3817d0d/volumes" Oct 29 22:05:14.544195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544165 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:05:14.544794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544781 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" Oct 29 22:05:14.544841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544795 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" Oct 29 22:05:14.544841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544821 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" Oct 29 22:05:14.544841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544826 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" Oct 29 22:05:14.544841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544836 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" Oct 29 22:05:14.544841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544841 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544852 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9b0d6af7-136d-4ce4-8777-ff93f1351825" containerName="revoke-keys" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544857 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0d6af7-136d-4ce4-8777-ff93f1351825" containerName="revoke-keys" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544962 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39396390-b94e-42fa-ae91-e837573e4009" containerName="istio-proxy" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544975 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e06dcf5-9c2a-4692-aa04-cfc2bcf05cd6" containerName="istio-proxy" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544983 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6111b3-741b-43c8-9bee-4368f3817d0d" containerName="istio-proxy" Oct 29 22:05:14.545018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.544991 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0d6af7-136d-4ce4-8777-ff93f1351825" containerName="revoke-keys" Oct 29 22:05:14.548858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.548839 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.569722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.569695 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:05:14.579378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.579357 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b16b28\"" Oct 29 22:05:14.579668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.579605 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b16b28-dockercfg-p4f2c\"" Oct 29 22:05:14.700101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.690383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.700101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.699533 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xd64\" (UniqueName: \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.750103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.749124 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:05:14.787100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.776031 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:14.787100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.778799 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:05:14.809184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.806114 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xd64\" (UniqueName: \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.809184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.806437 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.809184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.807555 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.834439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.834412 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xd64\" (UniqueName: \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.861495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.861470 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:14.907993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.907960 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv4mg\" (UniqueName: \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:14.908191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:14.908094 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.012316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.011117 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv4mg\" (UniqueName: \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.012316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.011188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.012316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.012126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.038289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.038258 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv4mg\" (UniqueName: \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\") pod \"payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.130769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.130689 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:15.195058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.194603 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:05:15.206869 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:15.205216 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb644c2_f3ff_4897_8dde_ce88159d7a9c.slice/crio-98f2625350896e1bde786fd700ae5e7b4aa7440dc9775f1a90e000d6ef231ec0 WatchSource:0}: Error finding container 98f2625350896e1bde786fd700ae5e7b4aa7440dc9775f1a90e000d6ef231ec0: Status 404 returned error can't find the container with id 98f2625350896e1bde786fd700ae5e7b4aa7440dc9775f1a90e000d6ef231ec0 Oct 29 22:05:15.457445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:15.457418 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:05:15.460616 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:15.460590 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fa0ea7_497e_4ece_a771_bbf66bbb8874.slice/crio-b84729fe507b7b05c1bd421570d4ef9cc71e9d64624e6041f2d4fd829c7606f3 WatchSource:0}: Error finding container b84729fe507b7b05c1bd421570d4ef9cc71e9d64624e6041f2d4fd829c7606f3: Status 404 returned error can't find the container with id b84729fe507b7b05c1bd421570d4ef9cc71e9d64624e6041f2d4fd829c7606f3 Oct 29 22:05:16.013484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.013448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" event={"ID":"c4fa0ea7-497e-4ece-a771-bbf66bbb8874","Type":"ContainerStarted","Data":"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e"} Oct 29 22:05:16.013949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.013514 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:16.013949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.013539 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" event={"ID":"c4fa0ea7-497e-4ece-a771-bbf66bbb8874","Type":"ContainerStarted","Data":"b84729fe507b7b05c1bd421570d4ef9cc71e9d64624e6041f2d4fd829c7606f3"} Oct 29 22:05:16.015131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.015103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" event={"ID":"8eb644c2-f3ff-4897-8dde-ce88159d7a9c","Type":"ContainerStarted","Data":"bc604bc3f04d00035ca266f96b38ebf660e66396458bc552737dd76afebe1855"} Oct 29 22:05:16.015239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.015153 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" event={"ID":"8eb644c2-f3ff-4897-8dde-ce88159d7a9c","Type":"ContainerStarted","Data":"98f2625350896e1bde786fd700ae5e7b4aa7440dc9775f1a90e000d6ef231ec0"} Oct 29 22:05:16.015239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.015210 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:16.065470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.065412 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" podStartSLOduration=2.065396839 podStartE2EDuration="2.065396839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:16.057534067 +0000 UTC m=+1377.799464743" watchObservedRunningTime="2026-10-29 22:05:16.065396839 +0000 UTC m=+1377.807327512" Oct 29 22:05:16.120119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.116687 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" podStartSLOduration=2.116665842 podStartE2EDuration="2.116665842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:16.105829693 +0000 UTC m=+1377.847760369" watchObservedRunningTime="2026-10-29 22:05:16.116665842 +0000 UTC m=+1377.858596516" Oct 29 22:05:16.893748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:16.893716 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:05:18.077049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.077016 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:05:18.081641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.081618 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.093800 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zk2n\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.093900 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.093989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094068 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094147 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094296 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094367 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094424 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.094485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.094474 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.105853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.105830 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:05:18.113950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.113926 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5bc531-openshift-default-dockercfg-w8hxt\"" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.199771 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.199841 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.199915 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.199968 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.200037 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.200117 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.200169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.200253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zk2n\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.200358 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.201453 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.201710 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.201931 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.202190 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.202800 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.248102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.245826 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.248102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.247357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zk2n\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.260097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.248752 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.260097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.258388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\") pod \"e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.425121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.424999 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5bc531-openshift-default-dockercfg-w8hxt\"" Oct 29 22:05:18.430187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.430164 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:18.773848 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:18.773796 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3251f60e_47f1_4fa7_abe3_370ed29d9549.slice/crio-320ac2ad1b9680880852d6e2ba13f2d9f54463f3f290c8edc90ee88cd0a84deb WatchSource:0}: Error finding container 320ac2ad1b9680880852d6e2ba13f2d9f54463f3f290c8edc90ee88cd0a84deb: Status 404 returned error can't find the container with id 320ac2ad1b9680880852d6e2ba13f2d9f54463f3f290c8edc90ee88cd0a84deb Oct 29 22:05:18.774430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.774405 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:05:18.777460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.777424 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:18.777546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.777491 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:18.777546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:18.777523 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:19.043832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.043746 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerStarted","Data":"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3"} Oct 29 22:05:19.043832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.043812 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerStarted","Data":"320ac2ad1b9680880852d6e2ba13f2d9f54463f3f290c8edc90ee88cd0a84deb"} Oct 29 22:05:19.094247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.094189 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podStartSLOduration=1.094176663 podStartE2EDuration="1.094176663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:19.091438088 +0000 UTC m=+1380.833368783" watchObservedRunningTime="2026-10-29 22:05:19.094176663 +0000 UTC m=+1380.836107392" Oct 29 22:05:19.430613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.430536 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:19.436426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.436404 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:19.875683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.875650 4543 scope.go:122] "RemoveContainer" containerID="ee31804f3f447b55028cb114d585025175b749ac99588cee00581bc35df583ef" Oct 29 22:05:19.888357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.888142 4543 scope.go:122] "RemoveContainer" containerID="e82e3a7a5017cb2165db0ee1a8ab6d98e47f45865211f0d40bec38dc0ba7c2be" Oct 29 22:05:19.898595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.898578 4543 scope.go:122] "RemoveContainer" containerID="3aefc17de225d12a960947c2cbf2b745f1136c49265fa3a3ae40ee76776f0868" Oct 29 22:05:19.908440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.908420 4543 scope.go:122] "RemoveContainer" containerID="6927efcdc104433b3f50c970d96e76c0c9dede3026077eb6c60f8715768b717e" Oct 29 22:05:19.917308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.917291 4543 scope.go:122] "RemoveContainer" containerID="7de752b1edf8884194bb78a93a538d79395f4e1a96b1ca41ede3c9ac1c695342" Oct 29 22:05:19.925971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.925957 4543 scope.go:122] "RemoveContainer" containerID="f577b86be85c3323ccad63e9de3c1473ce6cf1fba7e08143a19160994471c8c8" Oct 29 22:05:19.935117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.935103 4543 scope.go:122] "RemoveContainer" containerID="e442d886d7a2a92f71c3cd2db41aa76c0f2e480b27645c443633ffeec03eede4" Oct 29 22:05:19.944144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:19.944124 4543 scope.go:122] "RemoveContainer" containerID="6efb499964fd0ed53ad85148cd975d8067f5cef24c89290f8626117b69848942" Oct 29 22:05:20.055756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:20.055735 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:20.056804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:20.056780 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:05:23.479401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.479367 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:05:23.486328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.486309 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.506584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.506553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:05:23.511912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.511877 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5bc531\"" Oct 29 22:05:23.512035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.511954 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5bc531-dockercfg-n7pqk\"" Oct 29 22:05:23.608101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.601585 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzs2\" (UniqueName: \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.608101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.601659 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.663834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.663786 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:05:23.669469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.669205 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:05:23.669469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.669320 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.705117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.704513 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzs2\" (UniqueName: \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.705117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.704594 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.705507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.705480 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.734307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.734191 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzs2\" (UniqueName: \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\") pod \"payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.801589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.801558 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:23.816634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.816590 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.816858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.816836 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5dn\" (UniqueName: \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.919823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.919779 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5dn\" (UniqueName: \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.920018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.919997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.921282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.921253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:23.965718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:23.965659 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5dn\" (UniqueName: \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\") pod \"payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:24.003589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:24.003490 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:24.146516 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:24.146488 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2dbc2c_36be_4570_ae5b_354a8fc445ca.slice/crio-e0fc1cfe369d07da6e4cd35916b6704f1dbaa03a4217ab1e5bf6502d53d247cf WatchSource:0}: Error finding container e0fc1cfe369d07da6e4cd35916b6704f1dbaa03a4217ab1e5bf6502d53d247cf: Status 404 returned error can't find the container with id e0fc1cfe369d07da6e4cd35916b6704f1dbaa03a4217ab1e5bf6502d53d247cf Oct 29 22:05:24.149089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:24.149048 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:05:24.350138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:24.350114 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:05:24.350479 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:24.350457 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c3d7b2_7ce8_436e_b970_ca60ad9446cb.slice/crio-7665e6f5ece27e354a0691a401f59a7b2d5367da940efce2749c3618c0fb903a WatchSource:0}: Error finding container 7665e6f5ece27e354a0691a401f59a7b2d5367da940efce2749c3618c0fb903a: Status 404 returned error can't find the container with id 7665e6f5ece27e354a0691a401f59a7b2d5367da940efce2749c3618c0fb903a Oct 29 22:05:25.097226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.097192 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" event={"ID":"8d2dbc2c-36be-4570-ae5b-354a8fc445ca","Type":"ContainerStarted","Data":"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18"} Oct 29 22:05:25.097632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.097254 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" event={"ID":"8d2dbc2c-36be-4570-ae5b-354a8fc445ca","Type":"ContainerStarted","Data":"e0fc1cfe369d07da6e4cd35916b6704f1dbaa03a4217ab1e5bf6502d53d247cf"} Oct 29 22:05:25.097632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.097307 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:25.098969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.098940 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" event={"ID":"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb","Type":"ContainerStarted","Data":"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be"} Oct 29 22:05:25.099107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.098991 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" event={"ID":"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb","Type":"ContainerStarted","Data":"7665e6f5ece27e354a0691a401f59a7b2d5367da940efce2749c3618c0fb903a"} Oct 29 22:05:25.099107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.099013 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:25.149165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.149124 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" podStartSLOduration=2.149111556 podStartE2EDuration="2.149111556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:25.146561069 +0000 UTC m=+1386.888491772" watchObservedRunningTime="2026-10-29 22:05:25.149111556 +0000 UTC m=+1386.891042229" Oct 29 22:05:25.200437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:25.200386 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" podStartSLOduration=2.200370348 podStartE2EDuration="2.200370348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:25.196162448 +0000 UTC m=+1386.938093148" watchObservedRunningTime="2026-10-29 22:05:25.200370348 +0000 UTC m=+1386.942301021" Oct 29 22:05:27.027555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:27.027526 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:05:27.028128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:27.027594 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:05:27.039139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:27.039116 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc"] Oct 29 22:05:27.052395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:27.052373 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16b9892-vgcmc"] Oct 29 22:05:27.961278 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:27.961242 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:05:28.388455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:28.388378 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ec384c-4df2-4802-a5a1-025d107be768" path="/var/lib/kubelet/pods/43ec384c-4df2-4802-a5a1-025d107be768/volumes" Oct 29 22:05:31.056704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:31.056498 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr"] Oct 29 22:05:31.062406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:31.062380 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e129c5a-h9vlr"] Oct 29 22:05:31.668769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:31.668732 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q"] Oct 29 22:05:31.680178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:31.680148 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9f8c0-kx52q"] Oct 29 22:05:32.387657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:32.387626 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add88d61-d8a2-4e4c-8f3a-746a5eacb7bc" path="/var/lib/kubelet/pods/add88d61-d8a2-4e4c-8f3a-746a5eacb7bc/volumes" Oct 29 22:05:32.388109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:32.388094 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f3892d-ad84-4390-b4e7-584f6387a591" path="/var/lib/kubelet/pods/d5f3892d-ad84-4390-b4e7-584f6387a591/volumes" Oct 29 22:05:36.111392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:36.111363 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:05:36.111797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:36.111446 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:05:37.825510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:37.825474 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8"] Oct 29 22:05:37.838701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:37.838671 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-296d19-sz5m8"] Oct 29 22:05:38.386124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:38.386092 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d7a5bc-4844-49f0-9ecb-0e65c064e5b7" path="/var/lib/kubelet/pods/78d7a5bc-4844-49f0-9ecb-0e65c064e5b7/volumes" Oct 29 22:05:41.700295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.700261 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:05:41.704657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.704641 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:41.723329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.723310 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a20be1\"" Oct 29 22:05:41.741949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.741924 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:05:41.818098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.817898 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a20be1-dockercfg-6stn9\"" Oct 29 22:05:41.838652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.838605 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:41.884439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.884391 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rp4\" (UniqueName: \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:41.889383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.889351 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:05:41.895869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.895825 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:41.896437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.896417 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:05:41.986369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.986341 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76vb\" (UniqueName: \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:41.986574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.986382 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rp4\" (UniqueName: \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:41.986647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.986622 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:41.986749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.986736 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:41.987518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:41.987496 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:42.013127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.013053 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rp4\" (UniqueName: \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:42.028392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.028364 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:42.064212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.064180 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:05:42.088960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.088923 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.096328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.095211 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:05:42.099503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.099477 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76vb\" (UniqueName: \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:42.100993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.099964 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:42.101140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.101020 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:42.110739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.110714 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9bb81b-openshift-default-dockercfg-jzk4t\"" Oct 29 22:05:42.165562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.165541 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76vb\" (UniqueName: \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\") pod \"payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:42.203595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.203545 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.203762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.203692 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.203833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.203822 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.203938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.203919 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.203995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.203955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vsq\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.204048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.204036 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.204118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.204061 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.204200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.204180 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.204259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.204210 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.220963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.220928 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355408 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vsq\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355530 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355665 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355903 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.355950 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.356047 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.361097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.356937 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.450964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.370274 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.450964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.371337 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.450964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.379365 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.450964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.388029 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.450964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.447226 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.452933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.452027 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.538309 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:42.523379 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76b4bf9_2c4b_4030_b87c_11335fe04532.slice/crio-f8608520e94c7c7ce65b7b73605436eef9256ecd8f7b91305bc6738847194ae5 WatchSource:0}: Error finding container f8608520e94c7c7ce65b7b73605436eef9256ecd8f7b91305bc6738847194ae5: Status 404 returned error can't find the container with id f8608520e94c7c7ce65b7b73605436eef9256ecd8f7b91305bc6738847194ae5 Oct 29 22:05:42.541717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.541519 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.543376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.543279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vsq\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\") pod \"e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:42.553116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.553097 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:05:42.746029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.746005 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:05:42.746655 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:42.746631 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d66cb9_ef44_490f_a9f7_0bbe7bee68f4.slice/crio-fed7bc7d97bf6e362aed026a3dcf7eb14cbb36ffc96f11209237e9ed981eebd1 WatchSource:0}: Error finding container fed7bc7d97bf6e362aed026a3dcf7eb14cbb36ffc96f11209237e9ed981eebd1: Status 404 returned error can't find the container with id fed7bc7d97bf6e362aed026a3dcf7eb14cbb36ffc96f11209237e9ed981eebd1 Oct 29 22:05:42.765028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:42.765005 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:43.097303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.097274 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:05:43.098745 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:43.098720 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dea599f_9164_4a38_b80d_0de739784091.slice/crio-c1513f90b1e6fee53e2495681748bb58599e5175e175d21564c9098845627dc5 WatchSource:0}: Error finding container c1513f90b1e6fee53e2495681748bb58599e5175e175d21564c9098845627dc5: Status 404 returned error can't find the container with id c1513f90b1e6fee53e2495681748bb58599e5175e175d21564c9098845627dc5 Oct 29 22:05:43.101291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.101257 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:43.101401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.101333 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:43.101401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.101375 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:05:43.275139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.275103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" event={"ID":"c76b4bf9-2c4b-4030-b87c-11335fe04532","Type":"ContainerStarted","Data":"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84"} Oct 29 22:05:43.275339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.275164 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" event={"ID":"c76b4bf9-2c4b-4030-b87c-11335fe04532","Type":"ContainerStarted","Data":"f8608520e94c7c7ce65b7b73605436eef9256ecd8f7b91305bc6738847194ae5"} Oct 29 22:05:43.275339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.275185 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:43.277039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.277004 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerStarted","Data":"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0"} Oct 29 22:05:43.277201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.277063 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerStarted","Data":"c1513f90b1e6fee53e2495681748bb58599e5175e175d21564c9098845627dc5"} Oct 29 22:05:43.278782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.278762 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" event={"ID":"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4","Type":"ContainerStarted","Data":"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c"} Oct 29 22:05:43.278883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.278799 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" event={"ID":"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4","Type":"ContainerStarted","Data":"fed7bc7d97bf6e362aed026a3dcf7eb14cbb36ffc96f11209237e9ed981eebd1"} Oct 29 22:05:43.278883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.278840 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:43.322311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.322264 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" podStartSLOduration=2.322250244 podStartE2EDuration="2.322250244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:43.319677667 +0000 UTC m=+1405.061608352" watchObservedRunningTime="2026-10-29 22:05:43.322250244 +0000 UTC m=+1405.064180927" Oct 29 22:05:43.363820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.363740 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" podStartSLOduration=2.363725718 podStartE2EDuration="2.363725718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:43.36336036 +0000 UTC m=+1405.105291034" watchObservedRunningTime="2026-10-29 22:05:43.363725718 +0000 UTC m=+1405.105656391" Oct 29 22:05:43.417481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.417432 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podStartSLOduration=1.417415708 podStartE2EDuration="1.417415708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:43.412346903 +0000 UTC m=+1405.154277578" watchObservedRunningTime="2026-10-29 22:05:43.417415708 +0000 UTC m=+1405.159346381" Oct 29 22:05:43.765836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:43.765807 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:44.770204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:44.770177 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:45.294317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:45.294287 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:45.295158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:45.295136 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:05:46.462058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.462021 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:05:46.462572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.462267 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" podUID="317e9673-f09b-4719-972a-db8ac68a234f" containerName="payload-processing" containerID="cri-o://1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26" gracePeriod=30 Oct 29 22:05:46.741127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.741094 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:05:46.827144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.827057 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\" (UniqueName: \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\") pod \"317e9673-f09b-4719-972a-db8ac68a234f\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " Oct 29 22:05:46.830160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.830130 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\") pod \"317e9673-f09b-4719-972a-db8ac68a234f\" (UID: \"317e9673-f09b-4719-972a-db8ac68a234f\") " Oct 29 22:05:46.831847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.831819 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume" pod "317e9673-f09b-4719-972a-db8ac68a234f" (UID: "317e9673-f09b-4719-972a-db8ac68a234f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:46.834285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.834262 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317e9673-f09b-4719-972a-db8ac68a234f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:46.885856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.885814 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz" pod "317e9673-f09b-4719-972a-db8ac68a234f" (UID: "317e9673-f09b-4719-972a-db8ac68a234f"). InnerVolumeSpecName "kube-api-access-hvrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:46.940100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:46.937427 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrmz\" (UniqueName: \"kubernetes.io/projected/317e9673-f09b-4719-972a-db8ac68a234f-kube-api-access-hvrmz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:05:47.309163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.309127 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6"] Oct 29 22:05:47.315340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.315313 4543 generic.go:356] "Generic (PLEG): container finished" podID="317e9673-f09b-4719-972a-db8ac68a234f" containerID="1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26" exitCode=0 Oct 29 22:05:47.315478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.315382 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" Oct 29 22:05:47.315478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.315395 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" event={"ID":"317e9673-f09b-4719-972a-db8ac68a234f","Type":"ContainerDied","Data":"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26"} Oct 29 22:05:47.315571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.315477 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp" event={"ID":"317e9673-f09b-4719-972a-db8ac68a234f","Type":"ContainerDied","Data":"92ba1bc603162bf8ce10f9b47c96a6e5f1d1d5eeccc1cc4656d3118cf374d3a2"} Oct 29 22:05:47.315571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.315505 4543 scope.go:122] "RemoveContainer" containerID="1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26" Oct 29 22:05:47.323201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.323178 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad8e69-s25l6"] Oct 29 22:05:47.327997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.327979 4543 scope.go:122] "RemoveContainer" containerID="1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26" Oct 29 22:05:47.328405 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:47.328340 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26\": container with ID starting with 1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26 not found: ID does not exist" containerID="1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26" Oct 29 22:05:47.328405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.328373 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26"} err="failed to get container status \"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26\": rpc error: code = NotFound desc = could not find container \"1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26\": container with ID starting with 1243bcffb7745793bdf70055764347c0759f960a24c63e3cc47f5baf13ae1c26 not found: ID does not exist" Oct 29 22:05:47.377976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.376274 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:05:47.403101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:47.383480 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad8e69-f58df9c86-rv9lp"] Oct 29 22:05:48.386904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:48.386874 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317e9673-f09b-4719-972a-db8ac68a234f" path="/var/lib/kubelet/pods/317e9673-f09b-4719-972a-db8ac68a234f/volumes" Oct 29 22:05:48.387481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:48.387463 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0d6af7-136d-4ce4-8777-ff93f1351825" path="/var/lib/kubelet/pods/9b0d6af7-136d-4ce4-8777-ff93f1351825/volumes" Oct 29 22:05:54.289991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:54.289962 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:05:54.290480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:54.290028 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:05:55.222925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.222889 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:05:55.224121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.224099 4543 cpu_manager.go:414] "containerMap: removing container" podUID="317e9673-f09b-4719-972a-db8ac68a234f" containerName="payload-processing" Oct 29 22:05:55.224184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.224125 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e9673-f09b-4719-972a-db8ac68a234f" containerName="payload-processing" Oct 29 22:05:55.224375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.224360 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="317e9673-f09b-4719-972a-db8ac68a234f" containerName="payload-processing" Oct 29 22:05:55.230247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.230217 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.244745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.244717 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:05:55.247279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.247261 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fd1d4e\"" Oct 29 22:05:55.252907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.252885 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fd1d4e-dockercfg-47wrk\"" Oct 29 22:05:55.290254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.290220 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5km\" (UniqueName: \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.290674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.290515 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.377179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.376218 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:05:55.455421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.455382 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.455830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.455806 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5km\" (UniqueName: \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.456680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.456659 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.459249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.459132 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.477150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.474373 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:05:55.504834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.499861 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5km\" (UniqueName: \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.557582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.557550 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzdz\" (UniqueName: \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.557768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.557598 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.561380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.561350 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:55.659756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.659716 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzdz\" (UniqueName: \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.660045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.660021 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.661258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.661231 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.686647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.686618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzdz\" (UniqueName: \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\") pod \"payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.799192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.799096 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:55.883456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:55.883424 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:05:55.886574 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:55.886543 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a0609e_332f_477e_b8d3_e3c24a7cd3b5.slice/crio-ee4c4573f3886f982282670390982f08a36bd59c46133ac93d7d6a2fcca0a849 WatchSource:0}: Error finding container ee4c4573f3886f982282670390982f08a36bd59c46133ac93d7d6a2fcca0a849: Status 404 returned error can't find the container with id ee4c4573f3886f982282670390982f08a36bd59c46133ac93d7d6a2fcca0a849 Oct 29 22:05:56.006539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.006323 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:05:56.006714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.006575 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" containerID="cri-o://2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73" gracePeriod=30 Oct 29 22:05:56.125731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.125705 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:05:56.126479 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:05:56.126450 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35433c15_b08f_492e_a168_43789c3be5af.slice/crio-fab2d8f5702c082f6a3a2f5417514ca87c2af7a34d64706202be5266e90bc7be WatchSource:0}: Error finding container fab2d8f5702c082f6a3a2f5417514ca87c2af7a34d64706202be5266e90bc7be: Status 404 returned error can't find the container with id fab2d8f5702c082f6a3a2f5417514ca87c2af7a34d64706202be5266e90bc7be Oct 29 22:05:56.506761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.506474 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" event={"ID":"13a0609e-332f-477e-b8d3-e3c24a7cd3b5","Type":"ContainerStarted","Data":"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651"} Oct 29 22:05:56.506761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.506547 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:05:56.506761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.506570 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" event={"ID":"13a0609e-332f-477e-b8d3-e3c24a7cd3b5","Type":"ContainerStarted","Data":"ee4c4573f3886f982282670390982f08a36bd59c46133ac93d7d6a2fcca0a849"} Oct 29 22:05:56.508384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.508348 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" event={"ID":"35433c15-b08f-492e-a168-43789c3be5af","Type":"ContainerStarted","Data":"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485"} Oct 29 22:05:56.508516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.508393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" event={"ID":"35433c15-b08f-492e-a168-43789c3be5af","Type":"ContainerStarted","Data":"fab2d8f5702c082f6a3a2f5417514ca87c2af7a34d64706202be5266e90bc7be"} Oct 29 22:05:56.508516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.508414 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:05:56.554207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.554166 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" podStartSLOduration=1.5541518349999999 podStartE2EDuration="1.554151835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:56.549889163 +0000 UTC m=+1418.291819836" watchObservedRunningTime="2026-10-29 22:05:56.554151835 +0000 UTC m=+1418.296082508" Oct 29 22:05:56.598217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:05:56.598177 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" podStartSLOduration=1.598165208 podStartE2EDuration="1.598165208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:56.593949315 +0000 UTC m=+1418.335879988" watchObservedRunningTime="2026-10-29 22:05:56.598165208 +0000 UTC m=+1418.340095881" Oct 29 22:05:57.980381 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:05:57.980330 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:06:00.551547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.551510 4543 generic.go:356] "Generic (PLEG): container finished" podID="21dce852-8447-4004-989d-86ee0f6433a3" containerID="e75f2de3effe67c49540147ef2d166ddae21ebe2385e749e8327198ba4e33a04" exitCode=0 Oct 29 22:06:00.552109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.551556 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerDied","Data":"e75f2de3effe67c49540147ef2d166ddae21ebe2385e749e8327198ba4e33a04"} Oct 29 22:06:00.552109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.551998 4543 scope.go:122] "RemoveContainer" containerID="e75f2de3effe67c49540147ef2d166ddae21ebe2385e749e8327198ba4e33a04" Oct 29 22:06:00.553667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.553631 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:00.553775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.553705 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:00.553775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.553737 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:00.705591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:00.705558 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:06:01.474961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.474930 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:06:01.508115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508067 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508167 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508210 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508250 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508277 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508461 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:01.508508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508495 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508554 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508582 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:01.508720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508656 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.508772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508750 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:01.508831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508777 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\") pod \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\" (UID: \"8b6665a2-7cb2-49dc-9a04-dccd722eccb1\") " Oct 29 22:06:01.509147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.508948 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:01.509283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509186 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:01.509587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509562 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.509700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509592 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.509700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509606 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.509700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509620 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.509700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.509632 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.511147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.511119 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "kube-api-access-pk8lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:01.511952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.511925 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:01.512052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.512036 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:01.512712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.512693 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token" pod "8b6665a2-7cb2-49dc-9a04-dccd722eccb1" (UID: "8b6665a2-7cb2-49dc-9a04-dccd722eccb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:01.586522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.586457 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerStarted","Data":"caff5cff29bcb76ca2b3786267cf0ba0eac0618b2cf99bbc77486950207fa38e"} Oct 29 22:06:01.589059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.589031 4543 generic.go:356] "Generic (PLEG): container finished" podID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerID="2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73" exitCode=0 Oct 29 22:06:01.589239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.589114 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerDied","Data":"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73"} Oct 29 22:06:01.589239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.589158 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" event={"ID":"8b6665a2-7cb2-49dc-9a04-dccd722eccb1","Type":"ContainerDied","Data":"554115ece6ba49e1770015ed1d393e7ee597344e81b7b49783db6709fb200b7c"} Oct 29 22:06:01.589239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.589188 4543 scope.go:122] "RemoveContainer" containerID="2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73" Oct 29 22:06:01.589391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.589364 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754" Oct 29 22:06:01.601141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.601117 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:06:01.602064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.602036 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:01.602242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.602126 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:01.602280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.602253 4543 scope.go:122] "RemoveContainer" containerID="73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa" Oct 29 22:06:01.611963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.611891 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.611963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.611918 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.611963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.611930 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.611963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.611941 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk8lc\" (UniqueName: \"kubernetes.io/projected/8b6665a2-7cb2-49dc-9a04-dccd722eccb1-kube-api-access-pk8lc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:01.616126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.616063 4543 scope.go:122] "RemoveContainer" containerID="2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73" Oct 29 22:06:01.616421 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:01.616392 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73\": container with ID starting with 2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73 not found: ID does not exist" containerID="2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73" Oct 29 22:06:01.616542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.616431 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73"} err="failed to get container status \"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73\": rpc error: code = NotFound desc = could not find container \"2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73\": container with ID starting with 2cfe248fda57a753b0695dd7ced33ad8bb89176a842dfc0c07ea0aab3b1afc73 not found: ID does not exist" Oct 29 22:06:01.616542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.616449 4543 scope.go:122] "RemoveContainer" containerID="73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa" Oct 29 22:06:01.616830 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:01.616764 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa\": container with ID starting with 73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa not found: ID does not exist" containerID="73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa" Oct 29 22:06:01.616904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.616841 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa"} err="failed to get container status \"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa\": rpc error: code = NotFound desc = could not find container \"73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa\": container with ID starting with 73afd23fed19b62fa435dcdb766001e839440204898c60a80171ad91873b14fa not found: ID does not exist" Oct 29 22:06:01.698829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.698751 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:06:01.702306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:01.702284 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad8e69-openshift-default-6c8c69555f-4w754"] Oct 29 22:06:02.386812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:02.386778 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" path="/var/lib/kubelet/pods/8b6665a2-7cb2-49dc-9a04-dccd722eccb1/volumes" Oct 29 22:06:02.600505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:02.600473 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:02.600904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:02.600522 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:03.601140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:03.601098 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:03.601642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:03.601160 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:04.600469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:04.600439 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:04.600650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:04.600488 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:05.600703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.600631 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:05.601643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.601611 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:05.995066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.995032 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:06:05.995881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.995858 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:05.995999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.995882 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:05.995999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.995941 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:05.995999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.995949 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:05.996216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.996144 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:05.999675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:05.999654 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.019536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.019512 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:06:06.027136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.027112 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-796d71\"" Oct 29 22:06:06.027285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.027213 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-796d71-dockercfg-kzmtv\"" Oct 29 22:06:06.096182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.096129 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:06:06.101109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.101042 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6665a2-7cb2-49dc-9a04-dccd722eccb1" containerName="istio-proxy" Oct 29 22:06:06.103767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.103236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6z6\" (UniqueName: \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.103767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.103514 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.104884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.104862 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.213687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.213647 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:06:06.223108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.216692 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbmk\" (UniqueName: \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.223108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.216738 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.223108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.216783 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.223108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.216927 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6z6\" (UniqueName: \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.223108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.218302 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.302147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.291746 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6z6\" (UniqueName: \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\") pod \"payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.317429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.317400 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:06.326454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.318907 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbmk\" (UniqueName: \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.326454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.324926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.326454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.326208 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.397251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.397142 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbmk\" (UniqueName: \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\") pod \"payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.516026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.515874 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:06.600893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.600576 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:06:06.600893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.600631 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:06:06.760646 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:06.760604 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8247e7_df02_4f5c_98e3_cc6343bc383d.slice/crio-85be86e83bb1b508f3a62ab01bf54e55f0cdbf239f5727b899ab8c18905dc8a3 WatchSource:0}: Error finding container 85be86e83bb1b508f3a62ab01bf54e55f0cdbf239f5727b899ab8c18905dc8a3: Status 404 returned error can't find the container with id 85be86e83bb1b508f3a62ab01bf54e55f0cdbf239f5727b899ab8c18905dc8a3 Oct 29 22:06:06.760859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.760838 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:06:06.902953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:06.902932 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:06:06.904118 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:06.904087 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef5ad51_dddd_4d46_b244_44da697c900c.slice/crio-bc660e6d8cc036bb7b4b5c205d4adeca10ac364568ebf127a86135b927771dd8 WatchSource:0}: Error finding container bc660e6d8cc036bb7b4b5c205d4adeca10ac364568ebf127a86135b927771dd8: Status 404 returned error can't find the container with id bc660e6d8cc036bb7b4b5c205d4adeca10ac364568ebf127a86135b927771dd8 Oct 29 22:06:07.521779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.521753 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:06:07.521964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.521817 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:06:07.608863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.608826 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:06:07.609431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.609373 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:06:07.610211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.610187 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:06:07.680742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.680708 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" event={"ID":"3ef5ad51-dddd-4d46-b244-44da697c900c","Type":"ContainerStarted","Data":"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6"} Oct 29 22:06:07.680919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.680768 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" event={"ID":"3ef5ad51-dddd-4d46-b244-44da697c900c","Type":"ContainerStarted","Data":"bc660e6d8cc036bb7b4b5c205d4adeca10ac364568ebf127a86135b927771dd8"} Oct 29 22:06:07.680919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.680802 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:07.682766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.682712 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" event={"ID":"ae8247e7-df02-4f5c-98e3-cc6343bc383d","Type":"ContainerStarted","Data":"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195"} Oct 29 22:06:07.682903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.682786 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" event={"ID":"ae8247e7-df02-4f5c-98e3-cc6343bc383d","Type":"ContainerStarted","Data":"85be86e83bb1b508f3a62ab01bf54e55f0cdbf239f5727b899ab8c18905dc8a3"} Oct 29 22:06:07.682995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.682970 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:07.728280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.728221 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" podStartSLOduration=1.728203052 podStartE2EDuration="1.728203052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:07.723757001 +0000 UTC m=+1429.465687673" watchObservedRunningTime="2026-10-29 22:06:07.728203052 +0000 UTC m=+1429.470133727" Oct 29 22:06:07.769293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:07.769234 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" podStartSLOduration=2.769219698 podStartE2EDuration="2.769219698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:07.766445274 +0000 UTC m=+1429.508375946" watchObservedRunningTime="2026-10-29 22:06:07.769219698 +0000 UTC m=+1429.511150371" Oct 29 22:06:08.044933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.044899 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4"] Oct 29 22:06:08.048310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.048293 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.065263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.065237 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:06:08.065384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.065329 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:06:08.070364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.070328 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4"] Oct 29 22:06:08.112251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.112197 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:06:08.112399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.112272 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:06:08.185923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.185883 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.186159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.186131 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn8l\" (UniqueName: \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.186272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.186221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.286838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.286819 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.286948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.286933 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn8l\" (UniqueName: \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.287000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.286986 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.287755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.287730 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.289296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.289278 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.312463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.312413 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn8l\" (UniqueName: \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\") pod \"maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.362632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.362615 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:08.671203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.671175 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4"] Oct 29 22:06:08.671825 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:08.671794 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5aaad2_2405_4ea2_bb57_6f244f30ac38.slice/crio-8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038 WatchSource:0}: Error finding container 8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038: Status 404 returned error can't find the container with id 8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038 Oct 29 22:06:08.695754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:08.695676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" event={"ID":"1e5aaad2-2405-4ea2-bb57-6f244f30ac38","Type":"ContainerStarted","Data":"8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038"} Oct 29 22:06:09.705945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:09.705905 4543 generic.go:356] "Generic (PLEG): container finished" podID="1e5aaad2-2405-4ea2-bb57-6f244f30ac38" containerID="304c2f99c4df189046721ff1f6a0a36283750037909d596a2804538bfaefd422" exitCode=0 Oct 29 22:06:09.706316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:09.705986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" event={"ID":"1e5aaad2-2405-4ea2-bb57-6f244f30ac38","Type":"ContainerDied","Data":"304c2f99c4df189046721ff1f6a0a36283750037909d596a2804538bfaefd422"} Oct 29 22:06:10.207728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.207697 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 22:06:10.207919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.207896 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" containerID="cri-o://9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863" gracePeriod=30 Oct 29 22:06:10.219066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.219037 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 22:06:10.219314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.219286 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" containerID="cri-o://54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e" gracePeriod=30 Oct 29 22:06:10.223248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.223224 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:06:10.223411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.223393 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" containerID="cri-o://b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d" gracePeriod=30 Oct 29 22:06:10.895993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:10.895966 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:11.034159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.034084 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\") pod \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " Oct 29 22:06:11.034314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.034250 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\") pod \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " Oct 29 22:06:11.034410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.034387 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca" pod "1e5aaad2-2405-4ea2-bb57-6f244f30ac38" (UID: "1e5aaad2-2405-4ea2-bb57-6f244f30ac38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:11.034466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.034442 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\" (UniqueName: \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\") pod \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\" (UID: \"1e5aaad2-2405-4ea2-bb57-6f244f30ac38\") " Oct 29 22:06:11.034871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.034850 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:11.036476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.036456 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca" pod "1e5aaad2-2405-4ea2-bb57-6f244f30ac38" (UID: "1e5aaad2-2405-4ea2-bb57-6f244f30ac38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:06:11.036552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.036493 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l" pod "1e5aaad2-2405-4ea2-bb57-6f244f30ac38" (UID: "1e5aaad2-2405-4ea2-bb57-6f244f30ac38"). InnerVolumeSpecName "kube-api-access-qxn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:11.136639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.136608 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn8l\" (UniqueName: \"kubernetes.io/projected/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-kube-api-access-qxn8l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:11.136639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.136634 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5aaad2-2405-4ea2-bb57-6f244f30ac38-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:11.723659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.723628 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" Oct 29 22:06:11.723659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.723641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4" event={"ID":"1e5aaad2-2405-4ea2-bb57-6f244f30ac38","Type":"ContainerDied","Data":"8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038"} Oct 29 22:06:11.723893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:11.723683 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8344f52b793862d0e7e467d6affba598ee6bdc4f277225f1bc2b131806786038" Oct 29 22:06:14.491081 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:14.491054 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:06:14.528789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:14.528764 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:06:15.588040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.588010 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:06:15.645332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.645301 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 22:06:15.663824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.663800 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 22:06:15.701487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.701459 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.701735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.701714 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.701902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.701884 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702000 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702195 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702356 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702538 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702742 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.702893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.702880 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\") pod \"2e2e0c20-9efb-487c-ac56-72dac20dd253\" (UID: \"2e2e0c20-9efb-487c-ac56-72dac20dd253\") " Oct 29 22:06:15.703578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.703297 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.703914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.703890 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.703993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.703906 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.704352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.704326 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:15.704903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.704884 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.705046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.705027 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.705046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.705047 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.705276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.705061 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e0c20-9efb-487c-ac56-72dac20dd253-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.705276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.704985 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.707738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.707708 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.707738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.707711 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:15.708712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.708662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "kube-api-access-kxzv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.708879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.708851 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy" pod "2e2e0c20-9efb-487c-ac56-72dac20dd253" (UID: "2e2e0c20-9efb-487c-ac56-72dac20dd253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.797584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.797494 4543 generic.go:356] "Generic (PLEG): container finished" podID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerID="54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e" exitCode=0 Oct 29 22:06:15.797584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.797550 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" Oct 29 22:06:15.797788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.797581 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" event={"ID":"41e6fe73-41ef-41a4-8185-88a1f7802eaf","Type":"ContainerDied","Data":"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e"} Oct 29 22:06:15.797788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.797649 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg" event={"ID":"41e6fe73-41ef-41a4-8185-88a1f7802eaf","Type":"ContainerDied","Data":"fa74e4cdf2251b6346ddbeb3114cbab3db49c18f6234cb3ebf54850a70bb2315"} Oct 29 22:06:15.797788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.797681 4543 scope.go:122] "RemoveContainer" containerID="54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e" Oct 29 22:06:15.801324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.801185 4543 generic.go:356] "Generic (PLEG): container finished" podID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerID="b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d" exitCode=0 Oct 29 22:06:15.801324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.801281 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" event={"ID":"2e2e0c20-9efb-487c-ac56-72dac20dd253","Type":"ContainerDied","Data":"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d"} Oct 29 22:06:15.803306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803286 4543 generic.go:356] "Generic (PLEG): container finished" podID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerID="9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863" exitCode=0 Oct 29 22:06:15.803433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803418 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" Oct 29 22:06:15.803498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803450 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" event={"ID":"2e2e0c20-9efb-487c-ac56-72dac20dd253","Type":"ContainerDied","Data":"d2254275a8d95bda30b4d8625780b76ce573b3a4b5d26e580578c018f4203ab9"} Oct 29 22:06:15.803571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803510 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" event={"ID":"a8e97e32-b52b-440b-b0a8-61979fb1e1c7","Type":"ContainerDied","Data":"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863"} Oct 29 22:06:15.803571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj" event={"ID":"a8e97e32-b52b-440b-b0a8-61979fb1e1c7","Type":"ContainerDied","Data":"566e4b0a164a63684f94884e6d9aec87b3de021d8a279b57f150924cb3ab4a87"} Oct 29 22:06:15.803651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.803594 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn" Oct 29 22:06:15.805904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.805874 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.806009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.805974 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.806177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806066 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806121 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.806337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806232 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806305 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806350 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.806443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806380 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806407 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.806587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806444 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.806587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806507 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806914 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.806973 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807022 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807090 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807191 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807255 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\") pod \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\" (UID: \"41e6fe73-41ef-41a4-8185-88a1f7802eaf\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807311 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\") pod \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\" (UID: \"a8e97e32-b52b-440b-b0a8-61979fb1e1c7\") " Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808539 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808560 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808575 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808587 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzv6\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-kube-api-access-kxzv6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808599 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808612 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808628 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2e0c20-9efb-487c-ac56-72dac20dd253-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808641 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807312 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807346 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807496 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807655 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.807820 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808095 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:15.808847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.808411 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.810527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.809684 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.811542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.811515 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "kube-api-access-xzdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.812360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.812338 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.812671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.812649 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.812818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.812800 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:15.813423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.813378 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy" pod "a8e97e32-b52b-440b-b0a8-61979fb1e1c7" (UID: "a8e97e32-b52b-440b-b0a8-61979fb1e1c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:15.813423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.813393 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:15.815001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.814886 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887" pod "41e6fe73-41ef-41a4-8185-88a1f7802eaf" (UID: "41e6fe73-41ef-41a4-8185-88a1f7802eaf"). InnerVolumeSpecName "kube-api-access-dk887". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:15.815001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.814928 4543 scope.go:122] "RemoveContainer" containerID="54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e" Oct 29 22:06:15.815371 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:15.815342 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e\": container with ID starting with 54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e not found: ID does not exist" containerID="54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e" Oct 29 22:06:15.815441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.815383 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e"} err="failed to get container status \"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e\": rpc error: code = NotFound desc = could not find container \"54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e\": container with ID starting with 54ad07dd04bbed79e8fa9638113b024925af3ff52a09dd78924038aea344830e not found: ID does not exist" Oct 29 22:06:15.815441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.815402 4543 scope.go:122] "RemoveContainer" containerID="b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d" Oct 29 22:06:15.825839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.825816 4543 scope.go:122] "RemoveContainer" containerID="b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d" Oct 29 22:06:15.826121 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:15.826102 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d\": container with ID starting with b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d not found: ID does not exist" containerID="b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d" Oct 29 22:06:15.826181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.826128 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d"} err="failed to get container status \"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d\": rpc error: code = NotFound desc = could not find container \"b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d\": container with ID starting with b0294cf41db416c845c7bad718e2c12e2da0a13f6d8115b72293dd1e15385d6d not found: ID does not exist" Oct 29 22:06:15.826181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.826142 4543 scope.go:122] "RemoveContainer" containerID="9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863" Oct 29 22:06:15.835669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.835653 4543 scope.go:122] "RemoveContainer" containerID="9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863" Oct 29 22:06:15.835914 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:15.835897 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863\": container with ID starting with 9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863 not found: ID does not exist" containerID="9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863" Oct 29 22:06:15.835949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.835920 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863"} err="failed to get container status \"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863\": rpc error: code = NotFound desc = could not find container \"9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863\": container with ID starting with 9e2ac46932a0d7f535fb931dfaf1f1b3971e71c602e9e939f18cb4c5f914f863 not found: ID does not exist" Oct 29 22:06:15.893314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.893282 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:06:15.899149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.899127 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-qgltn"] Oct 29 22:06:15.910684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910658 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910681 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910693 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910702 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910712 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk887\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-kube-api-access-dk887\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910725 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910741 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzdsx\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-kube-api-access-xzdsx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910749 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910758 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910766 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910775 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910783 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910791 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910798 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e97e32-b52b-440b-b0a8-61979fb1e1c7-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:15.910806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:15.910805 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e6fe73-41ef-41a4-8185-88a1f7802eaf-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:16.150620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.150596 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 22:06:16.163637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.163617 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-59tbg"] Oct 29 22:06:16.197734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.197712 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 22:06:16.202552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.202531 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-9vbbj"] Oct 29 22:06:16.388011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.387950 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" path="/var/lib/kubelet/pods/2e2e0c20-9efb-487c-ac56-72dac20dd253/volumes" Oct 29 22:06:16.388473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.388457 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" path="/var/lib/kubelet/pods/41e6fe73-41ef-41a4-8185-88a1f7802eaf/volumes" Oct 29 22:06:16.388938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:16.388924 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" path="/var/lib/kubelet/pods/a8e97e32-b52b-440b-b0a8-61979fb1e1c7/volumes" Oct 29 22:06:18.218685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.218651 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 22:06:18.219187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.218908 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" containerID="cri-o://956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec" gracePeriod=30 Oct 29 22:06:18.233146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.233066 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 22:06:18.233427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.233401 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" containerID="cri-o://c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc" gracePeriod=30 Oct 29 22:06:18.237526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.237503 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:06:18.237739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.237713 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" containerID="cri-o://61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02" gracePeriod=30 Oct 29 22:06:18.627457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.627363 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:06:18.661259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.661234 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:06:18.698557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.698534 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:06:18.698857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:18.698835 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:06:19.491796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.491759 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" podUID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" containerName="payload-processing" containerID="cri-o://75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e" gracePeriod=25 Oct 29 22:06:19.529687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.529639 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" podUID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" containerName="payload-pre-processing" containerID="cri-o://bc604bc3f04d00035ca266f96b38ebf660e66396458bc552737dd76afebe1855" gracePeriod=25 Oct 29 22:06:19.845212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.845179 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:06:19.854679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.854646 4543 generic.go:356] "Generic (PLEG): container finished" podID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" containerID="75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e" exitCode=0 Oct 29 22:06:19.854844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.854782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" event={"ID":"c4fa0ea7-497e-4ece-a771-bbf66bbb8874","Type":"ContainerDied","Data":"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e"} Oct 29 22:06:19.854844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.854836 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" event={"ID":"c4fa0ea7-497e-4ece-a771-bbf66bbb8874","Type":"ContainerDied","Data":"b84729fe507b7b05c1bd421570d4ef9cc71e9d64624e6041f2d4fd829c7606f3"} Oct 29 22:06:19.854965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.854864 4543 scope.go:122] "RemoveContainer" containerID="75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e" Oct 29 22:06:19.855017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.854997 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6" Oct 29 22:06:19.871535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.871504 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\") pod \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " Oct 29 22:06:19.871671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.871559 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\" (UniqueName: \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\") pod \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\" (UID: \"c4fa0ea7-497e-4ece-a771-bbf66bbb8874\") " Oct 29 22:06:19.871671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.871644 4543 generic.go:356] "Generic (PLEG): container finished" podID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" containerID="bc604bc3f04d00035ca266f96b38ebf660e66396458bc552737dd76afebe1855" exitCode=0 Oct 29 22:06:19.871843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.871821 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" event={"ID":"8eb644c2-f3ff-4897-8dde-ce88159d7a9c","Type":"ContainerDied","Data":"bc604bc3f04d00035ca266f96b38ebf660e66396458bc552737dd76afebe1855"} Oct 29 22:06:19.872527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.872501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume" pod "c4fa0ea7-497e-4ece-a771-bbf66bbb8874" (UID: "c4fa0ea7-497e-4ece-a771-bbf66bbb8874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:19.878639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.878478 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg" pod "c4fa0ea7-497e-4ece-a771-bbf66bbb8874" (UID: "c4fa0ea7-497e-4ece-a771-bbf66bbb8874"). InnerVolumeSpecName "kube-api-access-bv4mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:19.887623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.887544 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:06:19.887623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.887615 4543 scope.go:122] "RemoveContainer" containerID="75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e" Oct 29 22:06:19.887928 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:19.887903 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e\": container with ID starting with 75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e not found: ID does not exist" containerID="75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e" Oct 29 22:06:19.887992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.887940 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e"} err="failed to get container status \"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e\": rpc error: code = NotFound desc = could not find container \"75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e\": container with ID starting with 75f06f53734a2a4f92e9b940e147c83c966a4de7156bd3992d32dca857e0862e not found: ID does not exist" Oct 29 22:06:19.973114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973049 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\") pod \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " Oct 29 22:06:19.973268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973184 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\" (UniqueName: \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\") pod \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\" (UID: \"8eb644c2-f3ff-4897-8dde-ce88159d7a9c\") " Oct 29 22:06:19.973375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973354 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume" pod "8eb644c2-f3ff-4897-8dde-ce88159d7a9c" (UID: "8eb644c2-f3ff-4897-8dde-ce88159d7a9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:19.973716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973701 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.973756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973720 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.973756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.973730 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv4mg\" (UniqueName: \"kubernetes.io/projected/c4fa0ea7-497e-4ece-a771-bbf66bbb8874-kube-api-access-bv4mg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.975485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:19.975464 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64" pod "8eb644c2-f3ff-4897-8dde-ce88159d7a9c" (UID: "8eb644c2-f3ff-4897-8dde-ce88159d7a9c"). InnerVolumeSpecName "kube-api-access-9xd64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:20.074581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.074524 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xd64\" (UniqueName: \"kubernetes.io/projected/8eb644c2-f3ff-4897-8dde-ce88159d7a9c-kube-api-access-9xd64\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:20.079468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.079448 4543 scope.go:122] "RemoveContainer" containerID="561e6f3d43d8182c830b406c596dd9f875069b07236620e044829bbdafcd57a0" Oct 29 22:06:20.090289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.090273 4543 scope.go:122] "RemoveContainer" containerID="4b681318313d008ab288ae04eea5cc72aae2869db6b68f78e32cae6b9a3567a7" Oct 29 22:06:20.100485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.100466 4543 scope.go:122] "RemoveContainer" containerID="96534d3267bc2488a70d5a64ead92eece23ddc8abec541e7614da03e88cf3441" Oct 29 22:06:20.111430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.111406 4543 scope.go:122] "RemoveContainer" containerID="2a160da2fb87539be964e5caeb7096c074c267d15539be8b77f2a7087e89c52d" Oct 29 22:06:20.204033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.204006 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:06:20.208709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.208683 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-5f44556ff6-whxs6"] Oct 29 22:06:20.313193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.313166 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:06:20.313293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.313211 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:06:20.386497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.386444 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" path="/var/lib/kubelet/pods/c4fa0ea7-497e-4ece-a771-bbf66bbb8874/volumes" Oct 29 22:06:20.879825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.879795 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" Oct 29 22:06:20.880343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.879818 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl" event={"ID":"8eb644c2-f3ff-4897-8dde-ce88159d7a9c","Type":"ContainerDied","Data":"98f2625350896e1bde786fd700ae5e7b4aa7440dc9775f1a90e000d6ef231ec0"} Oct 29 22:06:20.880343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.879930 4543 scope.go:122] "RemoveContainer" containerID="bc604bc3f04d00035ca266f96b38ebf660e66396458bc552737dd76afebe1855" Oct 29 22:06:20.938154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.938128 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:06:20.951208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:20.951187 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-5f66b5bf8c-rt8fl"] Oct 29 22:06:21.211158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:21.207163 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4"] Oct 29 22:06:21.211158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:21.210762 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4d79fa-tm8v4"] Oct 29 22:06:22.387399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:22.387367 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5aaad2-2405-4ea2-bb57-6f244f30ac38" path="/var/lib/kubelet/pods/1e5aaad2-2405-4ea2-bb57-6f244f30ac38/volumes" Oct 29 22:06:22.387800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:22.387762 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" path="/var/lib/kubelet/pods/8eb644c2-f3ff-4897-8dde-ce88159d7a9c/volumes" Oct 29 22:06:23.531240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.531211 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:06:23.589655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.589633 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 22:06:23.593478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.593459 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 22:06:23.623364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623336 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623403 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623433 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623518 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623626 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623668 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623805 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.623873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623837 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.623924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.623913 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\") pod \"7ade5319-70bd-458b-a7ff-667141033adf\" (UID: \"7ade5319-70bd-458b-a7ff-667141033adf\") " Oct 29 22:06:23.625425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.624634 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.625565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.624931 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.625565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.624693 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.625565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.624800 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:23.625881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.625859 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.626663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.626628 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "kube-api-access-2x46b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.627177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.627142 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:23.627660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.627640 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.628110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.627933 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" podUID="45456c4c-6916-4832-94e5-d4f5e7aa8192" containerName="payload-processing" containerID="cri-o://080ef44b52e9024fa7417a15be5ff5ec4f8611a5e3938943b2ba9a52a271ad93" gracePeriod=25 Oct 29 22:06:23.628658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.628636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy" pod "7ade5319-70bd-458b-a7ff-667141033adf" (UID: "7ade5319-70bd-458b-a7ff-667141033adf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.662589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.662549 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" podUID="4521277a-d81d-4450-a77d-d5139599f2fc" containerName="payload-pre-processing" containerID="cri-o://4db757d4bc38a242baf74ed0c81c5b3fc65c55f32dbb5caf2408e29e12b41758" gracePeriod=25 Oct 29 22:06:23.726239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726199 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726245 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726327 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726386 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726427 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726503 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726539 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726572 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726600 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726636 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.726752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726672 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726669 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.726752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726736 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726778 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726863 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726899 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\") pod \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\" (UID: \"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa\") " Oct 29 22:06:23.726991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726903 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.726991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.726934 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.727279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.727279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727158 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\") pod \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\" (UID: \"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078\") " Oct 29 22:06:23.727445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727422 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.727595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727570 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.727810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727780 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.728006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.727971 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.728253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.728378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728335 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.728378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728360 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728707 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728720 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728732 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728746 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728760 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x46b\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-kube-api-access-2x46b\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728773 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ade5319-70bd-458b-a7ff-667141033adf-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728786 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728799 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ade5319-70bd-458b-a7ff-667141033adf-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728813 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728826 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728838 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728851 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ade5319-70bd-458b-a7ff-667141033adf-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728863 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728875 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728887 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728901 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ade5319-70bd-458b-a7ff-667141033adf-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.728935 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.729050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.729030 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.730277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.729879 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.730277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.730105 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:23.730695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.730642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.730783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.730755 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.731758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.731719 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:23.732309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.732276 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:23.732401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.732384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz" pod "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" (UID: "c5c01e7d-527e-4ee6-9298-3a7d7bfb9078"). InnerVolumeSpecName "kube-api-access-94jpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.732511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.732494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl" pod "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" (UID: "2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa"). InnerVolumeSpecName "kube-api-access-xs2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830327 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830356 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830364 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830372 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830380 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830387 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830396 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2gl\" (UniqueName: \"kubernetes.io/projected/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-kube-api-access-xs2gl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830406 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94jpz\" (UniqueName: \"kubernetes.io/projected/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078-kube-api-access-94jpz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.833098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.830414 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:23.919272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.919224 4543 generic.go:356] "Generic (PLEG): container finished" podID="4521277a-d81d-4450-a77d-d5139599f2fc" containerID="4db757d4bc38a242baf74ed0c81c5b3fc65c55f32dbb5caf2408e29e12b41758" exitCode=0 Oct 29 22:06:23.919458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.919333 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" event={"ID":"4521277a-d81d-4450-a77d-d5139599f2fc","Type":"ContainerDied","Data":"4db757d4bc38a242baf74ed0c81c5b3fc65c55f32dbb5caf2408e29e12b41758"} Oct 29 22:06:23.925647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.925474 4543 generic.go:356] "Generic (PLEG): container finished" podID="7ade5319-70bd-458b-a7ff-667141033adf" containerID="61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02" exitCode=0 Oct 29 22:06:23.925966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.925901 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" event={"ID":"7ade5319-70bd-458b-a7ff-667141033adf","Type":"ContainerDied","Data":"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02"} Oct 29 22:06:23.926256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.926220 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" event={"ID":"7ade5319-70bd-458b-a7ff-667141033adf","Type":"ContainerDied","Data":"a28da471d9f1376118be718d051d6e7a59badc1c6dcd49686ee478124c821b7a"} Oct 29 22:06:23.926417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.926372 4543 scope.go:122] "RemoveContainer" containerID="61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02" Oct 29 22:06:23.926736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.926696 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5" Oct 29 22:06:23.935553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.935525 4543 generic.go:356] "Generic (PLEG): container finished" podID="45456c4c-6916-4832-94e5-d4f5e7aa8192" containerID="080ef44b52e9024fa7417a15be5ff5ec4f8611a5e3938943b2ba9a52a271ad93" exitCode=0 Oct 29 22:06:23.935672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.935654 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" event={"ID":"45456c4c-6916-4832-94e5-d4f5e7aa8192","Type":"ContainerDied","Data":"080ef44b52e9024fa7417a15be5ff5ec4f8611a5e3938943b2ba9a52a271ad93"} Oct 29 22:06:23.939258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.937728 4543 generic.go:356] "Generic (PLEG): container finished" podID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerID="c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc" exitCode=0 Oct 29 22:06:23.939258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.937881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerDied","Data":"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc"} Oct 29 22:06:23.939258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.937917 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" event={"ID":"2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa","Type":"ContainerDied","Data":"96d7dccf95eba0f7d4c3b52fda731dfcd3bf5bff8e4eeb373d7a36a0b705f421"} Oct 29 22:06:23.939258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.938010 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm" Oct 29 22:06:23.949135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.948555 4543 generic.go:356] "Generic (PLEG): container finished" podID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerID="956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec" exitCode=0 Oct 29 22:06:23.949135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.948600 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" event={"ID":"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078","Type":"ContainerDied","Data":"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec"} Oct 29 22:06:23.949135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.948635 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" event={"ID":"c5c01e7d-527e-4ee6-9298-3a7d7bfb9078","Type":"ContainerDied","Data":"b29a3e0983404adc87c9ae91e286ccadee3a8a61edc7e78bfce91d75c3813cea"} Oct 29 22:06:23.949135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.948717 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl" Oct 29 22:06:23.982711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.982687 4543 scope.go:122] "RemoveContainer" containerID="61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02" Oct 29 22:06:23.983123 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:23.983058 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02\": container with ID starting with 61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02 not found: ID does not exist" containerID="61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02" Oct 29 22:06:23.983123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.983119 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02"} err="failed to get container status \"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02\": rpc error: code = NotFound desc = could not find container \"61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02\": container with ID starting with 61afeed24da0a4c8112a10ff0f8f5e0117629115aadfad3167acb54569fdda02 not found: ID does not exist" Oct 29 22:06:23.983295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.983141 4543 scope.go:122] "RemoveContainer" containerID="c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc" Oct 29 22:06:23.996897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.996877 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:06:23.997652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:23.997631 4543 scope.go:122] "RemoveContainer" containerID="4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600" Oct 29 22:06:24.010907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.010884 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:06:24.018797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.018778 4543 scope.go:122] "RemoveContainer" containerID="c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc" Oct 29 22:06:24.020463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.019698 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-4knr5"] Oct 29 22:06:24.023558 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:24.022751 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc\": container with ID starting with c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc not found: ID does not exist" containerID="c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc" Oct 29 22:06:24.023558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.022789 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc"} err="failed to get container status \"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc\": rpc error: code = NotFound desc = could not find container \"c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc\": container with ID starting with c4a5e33986dda255d0faa815395be82d3f3be552bb4f092a027da71f18fcdffc not found: ID does not exist" Oct 29 22:06:24.023558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.022814 4543 scope.go:122] "RemoveContainer" containerID="4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600" Oct 29 22:06:24.028691 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:24.028630 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600\": container with ID starting with 4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600 not found: ID does not exist" containerID="4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600" Oct 29 22:06:24.028691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.028666 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600"} err="failed to get container status \"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600\": rpc error: code = NotFound desc = could not find container \"4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600\": container with ID starting with 4b60d968196350a64ac465157974f993fead017facd35ff419120881f6824600 not found: ID does not exist" Oct 29 22:06:24.028691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.028689 4543 scope.go:122] "RemoveContainer" containerID="956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec" Oct 29 22:06:24.040342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.040324 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:06:24.040724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.040707 4543 scope.go:122] "RemoveContainer" containerID="956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec" Oct 29 22:06:24.041005 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:24.040987 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec\": container with ID starting with 956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec not found: ID does not exist" containerID="956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec" Oct 29 22:06:24.041062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.041013 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec"} err="failed to get container status \"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec\": rpc error: code = NotFound desc = could not find container \"956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec\": container with ID starting with 956e3f6ddfe901b397d8cdb732c1448d3c0fa1d0809789fd2b6d5f92691f2aec not found: ID does not exist" Oct 29 22:06:24.074908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.074880 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 22:06:24.085629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.085604 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-t8tfm"] Oct 29 22:06:24.147615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.147582 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\" (UniqueName: \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\") pod \"4521277a-d81d-4450-a77d-d5139599f2fc\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " Oct 29 22:06:24.147852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.147834 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\" (UniqueName: \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\") pod \"45456c4c-6916-4832-94e5-d4f5e7aa8192\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " Oct 29 22:06:24.147902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.147876 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\") pod \"45456c4c-6916-4832-94e5-d4f5e7aa8192\" (UID: \"45456c4c-6916-4832-94e5-d4f5e7aa8192\") " Oct 29 22:06:24.147981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.147960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\") pod \"4521277a-d81d-4450-a77d-d5139599f2fc\" (UID: \"4521277a-d81d-4450-a77d-d5139599f2fc\") " Oct 29 22:06:24.148959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.148925 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume" pod "4521277a-d81d-4450-a77d-d5139599f2fc" (UID: "4521277a-d81d-4450-a77d-d5139599f2fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:24.149599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.149557 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume" pod "45456c4c-6916-4832-94e5-d4f5e7aa8192" (UID: "45456c4c-6916-4832-94e5-d4f5e7aa8192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:24.150374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.150348 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr" pod "4521277a-d81d-4450-a77d-d5139599f2fc" (UID: "4521277a-d81d-4450-a77d-d5139599f2fc"). InnerVolumeSpecName "kube-api-access-w2qwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:24.150445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.150365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2" pod "45456c4c-6916-4832-94e5-d4f5e7aa8192" (UID: "45456c4c-6916-4832-94e5-d4f5e7aa8192"). InnerVolumeSpecName "kube-api-access-7jdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:24.231293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.231245 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 22:06:24.235890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.235866 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-knwcl"] Oct 29 22:06:24.250536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.250515 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4521277a-d81d-4450-a77d-d5139599f2fc-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:24.250631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.250538 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qwr\" (UniqueName: \"kubernetes.io/projected/4521277a-d81d-4450-a77d-d5139599f2fc-kube-api-access-w2qwr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:24.250631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.250550 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jdg2\" (UniqueName: \"kubernetes.io/projected/45456c4c-6916-4832-94e5-d4f5e7aa8192-kube-api-access-7jdg2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:24.250631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.250558 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45456c4c-6916-4832-94e5-d4f5e7aa8192-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:24.388326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.388257 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" path="/var/lib/kubelet/pods/2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa/volumes" Oct 29 22:06:24.388784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.388769 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ade5319-70bd-458b-a7ff-667141033adf" path="/var/lib/kubelet/pods/7ade5319-70bd-458b-a7ff-667141033adf/volumes" Oct 29 22:06:24.389281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.389266 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" path="/var/lib/kubelet/pods/c5c01e7d-527e-4ee6-9298-3a7d7bfb9078/volumes" Oct 29 22:06:24.800718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.800684 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:06:24.801654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801636 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4521277a-d81d-4450-a77d-d5139599f2fc" containerName="payload-pre-processing" Oct 29 22:06:24.801654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801656 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4521277a-d81d-4450-a77d-d5139599f2fc" containerName="payload-pre-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801678 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801685 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801699 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801706 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801722 4543 cpu_manager.go:414] "containerMap: removing container" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801729 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801744 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45456c4c-6916-4832-94e5-d4f5e7aa8192" containerName="payload-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801751 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45456c4c-6916-4832-94e5-d4f5e7aa8192" containerName="payload-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801771 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" containerName="payload-pre-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801778 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" containerName="payload-pre-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801793 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801800 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801813 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801820 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801829 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801836 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801850 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" containerName="payload-processing" Oct 29 22:06:24.801853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801857 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" containerName="payload-processing" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801874 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1e5aaad2-2405-4ea2-bb57-6f244f30ac38" containerName="revoke-keys" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801881 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5aaad2-2405-4ea2-bb57-6f244f30ac38" containerName="revoke-keys" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801891 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.801897 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802017 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb644c2-f3ff-4897-8dde-ce88159d7a9c" containerName="payload-pre-processing" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802029 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ade5319-70bd-458b-a7ff-667141033adf" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802039 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c01e7d-527e-4ee6-9298-3a7d7bfb9078" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802047 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e97e32-b52b-440b-b0a8-61979fb1e1c7" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802055 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fa0ea7-497e-4ece-a771-bbf66bbb8874" containerName="payload-processing" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802064 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5aaad2-2405-4ea2-bb57-6f244f30ac38" containerName="revoke-keys" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802088 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802101 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4521277a-d81d-4450-a77d-d5139599f2fc" containerName="payload-pre-processing" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802111 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9d89d4-3e13-4eb6-ac8c-6861c936cfaa" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802119 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2e0c20-9efb-487c-ac56-72dac20dd253" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802127 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e6fe73-41ef-41a4-8185-88a1f7802eaf" containerName="istio-proxy" Oct 29 22:06:24.803195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.802138 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45456c4c-6916-4832-94e5-d4f5e7aa8192" containerName="payload-processing" Oct 29 22:06:24.814115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.814064 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.831630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.831605 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6153f8-openshift-default-dockercfg-kx8rw\"" Oct 29 22:06:24.833042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.832962 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:06:24.957114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.957091 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" Oct 29 22:06:24.957259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.957107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn" event={"ID":"45456c4c-6916-4832-94e5-d4f5e7aa8192","Type":"ContainerDied","Data":"a40a66f11289ca7bbf70a85ff88366f1f518f328a3dce9d4f25e4b4d433e5264"} Oct 29 22:06:24.957259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.957173 4543 scope.go:122] "RemoveContainer" containerID="080ef44b52e9024fa7417a15be5ff5ec4f8611a5e3938943b2ba9a52a271ad93" Oct 29 22:06:24.958538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958271 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958391 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6fd\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958575 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958662 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958753 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958796 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958839 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.958905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.958885 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:24.961309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.961289 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" Oct 29 22:06:24.961417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.961288 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9" event={"ID":"4521277a-d81d-4450-a77d-d5139599f2fc","Type":"ContainerDied","Data":"91979b1c6ef13a415efb9efa9f7a378284a9e0fa1da9678fa9a78af272175c73"} Oct 29 22:06:24.969015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:24.968998 4543 scope.go:122] "RemoveContainer" containerID="4db757d4bc38a242baf74ed0c81c5b3fc65c55f32dbb5caf2408e29e12b41758" Oct 29 22:06:25.015339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.015318 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:06:25.021161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.021144 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4d79fa-77577fcb98-cngqn"] Oct 29 22:06:25.060164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.059949 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.060442 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.060516 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.060609 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.060864 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6fd\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.061053 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.061142 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.061173 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.061203 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.061351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.061246 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.063994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.062449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.063994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.063441 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.063994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.063790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.064224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.064182 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.064454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.064435 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.069732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.069642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.083060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.079176 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4d79fa-5d74dbb9dc-zbwf9"] Oct 29 22:06:25.083060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.083010 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.114210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.114132 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6fd\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.118700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.118672 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\") pod \"e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.138392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.138356 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:25.548345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.548310 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:06:25.554239 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:25.554208 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1123aa_5079_4e61_bfdc_e670b00d78c9.slice/crio-493f6ff7d363d4a9b89268454860b9becac039036f1ddd2129a8b23b854a2325 WatchSource:0}: Error finding container 493f6ff7d363d4a9b89268454860b9becac039036f1ddd2129a8b23b854a2325: Status 404 returned error can't find the container with id 493f6ff7d363d4a9b89268454860b9becac039036f1ddd2129a8b23b854a2325 Oct 29 22:06:25.556875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.556830 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:25.556972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.556924 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:25.557022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.556978 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:25.973549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.973502 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerStarted","Data":"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b"} Oct 29 22:06:25.973964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:25.973610 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerStarted","Data":"493f6ff7d363d4a9b89268454860b9becac039036f1ddd2129a8b23b854a2325"} Oct 29 22:06:26.027043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.026994 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podStartSLOduration=2.026979993 podStartE2EDuration="2.026979993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:26.02534711 +0000 UTC m=+1447.767277784" watchObservedRunningTime="2026-10-29 22:06:26.026979993 +0000 UTC m=+1447.768910665" Oct 29 22:06:26.138843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.138775 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:26.145328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.145305 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:26.387604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.387515 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4521277a-d81d-4450-a77d-d5139599f2fc" path="/var/lib/kubelet/pods/4521277a-d81d-4450-a77d-d5139599f2fc/volumes" Oct 29 22:06:26.388016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.387995 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45456c4c-6916-4832-94e5-d4f5e7aa8192" path="/var/lib/kubelet/pods/45456c4c-6916-4832-94e5-d4f5e7aa8192/volumes" Oct 29 22:06:26.982751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.982722 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:26.983694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:26.983675 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:06:27.961680 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:27.961638 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:06:30.221177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.221140 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:06:30.221807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.221435 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" containerID="cri-o://2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0" gracePeriod=30 Oct 29 22:06:30.580157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.580067 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:06:30.585597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.585576 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.602884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.602864 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6153f8\"" Oct 29 22:06:30.603009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.602899 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6153f8-dockercfg-g6j4l\"" Oct 29 22:06:30.604448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.604428 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:06:30.708098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.691754 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf77t\" (UniqueName: \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.708098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.691983 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.724882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.724842 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:06:30.778058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.778025 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:06:30.778244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.778198 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:30.796099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.796050 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.796270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.796249 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf77t\" (UniqueName: \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.797884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.797855 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.823553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.823523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf77t\" (UniqueName: \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\") pod \"payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.873428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.873349 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh"] Oct 29 22:06:30.880880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.880851 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh"] Oct 29 22:06:30.881030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.881003 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.897911 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.897985 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvq9\" (UniqueName: \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.899518 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.900322 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.900499 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.900650 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:06:30.903122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.900715 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.999826 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:30.999994 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.000055 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvq9\" (UniqueName: \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.000114 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.000214 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.001624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.001261 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:31.027850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.027814 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvq9\" (UniqueName: \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\") pod \"payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:31.102064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.101906 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.102224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.102097 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.102224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.102193 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.102942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.102915 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.104945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.104919 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.114417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.114141 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:31.155265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.146330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\") pod \"maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.209832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.209802 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:31.326208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.325169 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:06:31.468638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.468606 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:06:31.469102 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:31.469058 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7a915d_a0f1_4d49_b2fe_6e32efaaf548.slice/crio-4bbd13dbbbebc512c63e2de6191aae913edf712b127624486abb9e910d1ee215 WatchSource:0}: Error finding container 4bbd13dbbbebc512c63e2de6191aae913edf712b127624486abb9e910d1ee215: Status 404 returned error can't find the container with id 4bbd13dbbbebc512c63e2de6191aae913edf712b127624486abb9e910d1ee215 Oct 29 22:06:31.575332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:31.575300 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh"] Oct 29 22:06:31.576735 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:31.576706 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf25e76b_2707_415f_9430_0830c7b22ba8.slice/crio-e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378 WatchSource:0}: Error finding container e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378: Status 404 returned error can't find the container with id e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378 Oct 29 22:06:32.023395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.023308 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" event={"ID":"af25e76b-2707-415f-9430-0830c7b22ba8","Type":"ContainerStarted","Data":"f82747d89f7a57b586730726bac1613b83946453394cedd234e6c3a62f165757"} Oct 29 22:06:32.023395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.023380 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" event={"ID":"af25e76b-2707-415f-9430-0830c7b22ba8","Type":"ContainerStarted","Data":"e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378"} Oct 29 22:06:32.025378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.025344 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" event={"ID":"e7e06426-c97d-47c2-b662-d3c235dd9706","Type":"ContainerStarted","Data":"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6"} Oct 29 22:06:32.025517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.025405 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" event={"ID":"e7e06426-c97d-47c2-b662-d3c235dd9706","Type":"ContainerStarted","Data":"2ba9cb1927a9fcfb5186f3191fecfcc160b0cda7e0c8864409ec1e7f67c73711"} Oct 29 22:06:32.025589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.025564 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:32.027393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.027369 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" event={"ID":"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548","Type":"ContainerStarted","Data":"f464780fb6a4a8ffe06ce9275d2a315bb9771c1688a77c12bf19811a79086516"} Oct 29 22:06:32.027484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.027411 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" event={"ID":"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548","Type":"ContainerStarted","Data":"4bbd13dbbbebc512c63e2de6191aae913edf712b127624486abb9e910d1ee215"} Oct 29 22:06:32.027484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.027442 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:32.082949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.082895 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" podStartSLOduration=2.08287893 podStartE2EDuration="2.08287893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:32.080781635 +0000 UTC m=+1453.822712311" watchObservedRunningTime="2026-10-29 22:06:32.08287893 +0000 UTC m=+1453.824809605" Oct 29 22:06:32.160371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.160308 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" podStartSLOduration=2.160293969 podStartE2EDuration="2.160293969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:32.126667666 +0000 UTC m=+1453.868598340" watchObservedRunningTime="2026-10-29 22:06:32.160293969 +0000 UTC m=+1453.902224641" Oct 29 22:06:32.181999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:32.181948 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" podStartSLOduration=2.181934667 podStartE2EDuration="2.181934667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:32.179511967 +0000 UTC m=+1453.921442639" watchObservedRunningTime="2026-10-29 22:06:32.181934667 +0000 UTC m=+1453.923865340" Oct 29 22:06:33.060867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.060829 4543 generic.go:356] "Generic (PLEG): container finished" podID="af25e76b-2707-415f-9430-0830c7b22ba8" containerID="f82747d89f7a57b586730726bac1613b83946453394cedd234e6c3a62f165757" exitCode=0 Oct 29 22:06:33.061345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.060915 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" event={"ID":"af25e76b-2707-415f-9430-0830c7b22ba8","Type":"ContainerDied","Data":"f82747d89f7a57b586730726bac1613b83946453394cedd234e6c3a62f165757"} Oct 29 22:06:33.106627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.106602 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf"] Oct 29 22:06:33.110015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.109989 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19d4826-nfspf"] Oct 29 22:06:33.224985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.224954 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:06:33.225174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:33.225005 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:06:34.236883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.236856 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:34.372697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.372630 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\") pod \"af25e76b-2707-415f-9430-0830c7b22ba8\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " Oct 29 22:06:34.372697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.372682 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\") pod \"af25e76b-2707-415f-9430-0830c7b22ba8\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " Oct 29 22:06:34.372880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.372749 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\") pod \"af25e76b-2707-415f-9430-0830c7b22ba8\" (UID: \"af25e76b-2707-415f-9430-0830c7b22ba8\") " Oct 29 22:06:34.373146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.373120 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca" pod "af25e76b-2707-415f-9430-0830c7b22ba8" (UID: "af25e76b-2707-415f-9430-0830c7b22ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:34.374980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.374953 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca" pod "af25e76b-2707-415f-9430-0830c7b22ba8" (UID: "af25e76b-2707-415f-9430-0830c7b22ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:06:34.375130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.374982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9" pod "af25e76b-2707-415f-9430-0830c7b22ba8" (UID: "af25e76b-2707-415f-9430-0830c7b22ba8"). InnerVolumeSpecName "kube-api-access-cckw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:34.389824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.389801 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904cc8ad-57db-424e-abea-e35c6ffd3c3d" path="/var/lib/kubelet/pods/904cc8ad-57db-424e-abea-e35c6ffd3c3d/volumes" Oct 29 22:06:34.474025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.474000 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/af25e76b-2707-415f-9430-0830c7b22ba8-kube-api-access-cckw9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:34.474025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.474024 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:34.474207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:34.474036 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af25e76b-2707-415f-9430-0830c7b22ba8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.079805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.079773 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" Oct 29 22:06:35.080008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.079814 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh" event={"ID":"af25e76b-2707-415f-9430-0830c7b22ba8","Type":"ContainerDied","Data":"e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378"} Oct 29 22:06:35.080008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.079861 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9cdb8d7b908ecfea0fd9e3ff49e21060d39c598b6842a3e2be7af44cf91b378" Oct 29 22:06:35.526624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.526600 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:06:35.590777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.590741 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.590970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.590815 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.590970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.590946 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.590996 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.591060 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.591136 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.591177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.591279 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.591379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.591353 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\") pod \"5b992505-27ea-4e99-8e66-4fabfd4e5522\" (UID: \"5b992505-27ea-4e99-8e66-4fabfd4e5522\") " Oct 29 22:06:35.600597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.594890 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:35.600597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.595131 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:35.600597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.600324 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:35.600864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.600607 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:35.603756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.603723 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:35.604016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.603986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:06:35.604243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.604214 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "kube-api-access-7dr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:35.608500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.608475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:06:35.609515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.609494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token" pod "5b992505-27ea-4e99-8e66-4fabfd4e5522" (UID: "5b992505-27ea-4e99-8e66-4fabfd4e5522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:35.695762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695729 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695765 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695781 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695793 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695805 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695819 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dr5l\" (UniqueName: \"kubernetes.io/projected/5b992505-27ea-4e99-8e66-4fabfd4e5522-kube-api-access-7dr5l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695834 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b992505-27ea-4e99-8e66-4fabfd4e5522-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695847 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:35.695894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:35.695858 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b992505-27ea-4e99-8e66-4fabfd4e5522-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:36.088440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.088402 4543 generic.go:356] "Generic (PLEG): container finished" podID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerID="2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0" exitCode=0 Oct 29 22:06:36.088639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.088515 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" Oct 29 22:06:36.088639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.088537 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerDied","Data":"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0"} Oct 29 22:06:36.088639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.088590 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg" event={"ID":"5b992505-27ea-4e99-8e66-4fabfd4e5522","Type":"ContainerDied","Data":"ad9394fe84bfe41cf6a39ed5e2d704be710ac78ea8cd183bb2bf32d2b90fe84e"} Oct 29 22:06:36.088639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.088613 4543 scope.go:122] "RemoveContainer" containerID="2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0" Oct 29 22:06:36.097197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.097175 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:06:36.097301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.097217 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:06:36.102222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.102198 4543 scope.go:122] "RemoveContainer" containerID="6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73" Oct 29 22:06:36.112430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.112411 4543 scope.go:122] "RemoveContainer" containerID="2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0" Oct 29 22:06:36.112716 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:36.112688 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0\": container with ID starting with 2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0 not found: ID does not exist" containerID="2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0" Oct 29 22:06:36.112817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.112727 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0"} err="failed to get container status \"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0\": rpc error: code = NotFound desc = could not find container \"2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0\": container with ID starting with 2d308394624c64cfd5b2623c51caa9ac12b752da5ae46cda155b0236b85effc0 not found: ID does not exist" Oct 29 22:06:36.112817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.112750 4543 scope.go:122] "RemoveContainer" containerID="6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73" Oct 29 22:06:36.113399 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:36.113063 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73\": container with ID starting with 6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73 not found: ID does not exist" containerID="6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73" Oct 29 22:06:36.113399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.113119 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73"} err="failed to get container status \"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73\": rpc error: code = NotFound desc = could not find container \"6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73\": container with ID starting with 6262d638172aca52957a2d8a874e8f977297243c193af2e10662c570600b8f73 not found: ID does not exist" Oct 29 22:06:36.161218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.161191 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:06:36.178004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.177975 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4d79fa-openshift-default-f86ddf8f7-vlnlg"] Oct 29 22:06:36.386599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:36.386535 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" path="/var/lib/kubelet/pods/5b992505-27ea-4e99-8e66-4fabfd4e5522/volumes" Oct 29 22:06:38.109531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.109496 4543 generic.go:356] "Generic (PLEG): container finished" podID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerID="11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3" exitCode=0 Oct 29 22:06:38.110115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.109573 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerDied","Data":"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3"} Oct 29 22:06:38.110115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.110004 4543 scope.go:122] "RemoveContainer" containerID="11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3" Oct 29 22:06:38.111865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.111831 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:38.111966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.111920 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:38.112005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:38.111964 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:06:39.123548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:39.123516 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerStarted","Data":"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494"} Oct 29 22:06:39.452160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:39.451130 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:06:39.452160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:39.451269 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:39.452160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:39.451315 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:40.453311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:40.453187 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:40.453311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:40.453236 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:41.433494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:41.433454 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:41.433806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:41.433776 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:41.516239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:41.516212 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:06:42.435102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:42.433462 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:42.435102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:42.433553 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:43.064847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:43.064818 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:06:43.065318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:43.064889 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:06:43.439490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:43.439404 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:43.439788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:43.439748 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:44.438100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:44.433448 4543 patch_prober.go:29] interesting pod/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:06:44.438100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:44.433513 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 29 22:06:45.435062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:45.435031 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:06:45.435525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:45.435503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:06:45.436245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:45.436222 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:06:46.466235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.466207 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:06:46.466873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.466857 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.466917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.466874 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.466917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.466889 4543 cpu_manager.go:414] "containerMap: removing container" podUID="af25e76b-2707-415f-9430-0830c7b22ba8" containerName="revoke-keys" Oct 29 22:06:46.466917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.466894 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af25e76b-2707-415f-9430-0830c7b22ba8" containerName="revoke-keys" Oct 29 22:06:46.467023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.467014 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.467058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.467029 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="af25e76b-2707-415f-9430-0830c7b22ba8" containerName="revoke-keys" Oct 29 22:06:46.470060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.470041 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.490481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.490455 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:06:46.499280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.499256 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b16b28\"" Oct 29 22:06:46.499448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.499429 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b16b28-dockercfg-xwlt9\"" Oct 29 22:06:46.516578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.516541 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" podUID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" containerName="payload-pre-processing" containerID="cri-o://200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4" gracePeriod=25 Oct 29 22:06:46.533632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.533608 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:06:46.561536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.561512 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:06:46.581239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.581216 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:06:46.581935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.581916 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.582031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.581937 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.582119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.582104 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b992505-27ea-4e99-8e66-4fabfd4e5522" containerName="istio-proxy" Oct 29 22:06:46.584513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.584494 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.585595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.585576 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.585673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.585635 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz4t\" (UniqueName: \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.606568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.606547 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:06:46.698285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.698251 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfrt\" (UniqueName: \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.698527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.698453 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.698652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.698557 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz4t\" (UniqueName: \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.698718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.698674 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.699718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.699691 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.729809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.729744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz4t\" (UniqueName: \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\") pod \"payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.789861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.789809 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:46.800699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.800663 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.800865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.800779 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfrt\" (UniqueName: \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.801888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.801867 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.916443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.849603 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:06:46.916443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.874039 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfrt\" (UniqueName: \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\") pod \"payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:46.945558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:46.945519 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:47.007701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.007667 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\" (UniqueName: \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\") pod \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " Oct 29 22:06:47.008189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.007874 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\") pod \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\" (UID: \"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d\") " Oct 29 22:06:47.008430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.008328 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume" pod "0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" (UID: "0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:47.008663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.008629 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:47.011891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.011854 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5" pod "0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" (UID: "0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d"). InnerVolumeSpecName "kube-api-access-xm7z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:47.110345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.110311 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7z5\" (UniqueName: \"kubernetes.io/projected/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d-kube-api-access-xm7z5\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:47.216343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.216314 4543 generic.go:356] "Generic (PLEG): container finished" podID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" containerID="200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4" exitCode=0 Oct 29 22:06:47.216514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.216359 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" Oct 29 22:06:47.216514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.216399 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" event={"ID":"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d","Type":"ContainerDied","Data":"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4"} Oct 29 22:06:47.216514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.216455 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d" event={"ID":"0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d","Type":"ContainerDied","Data":"be6d98ecf7d9d8ba2e7d197de6d4db62fdfc899eb0ae54f8c8936f59cee142b4"} Oct 29 22:06:47.216514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.216484 4543 scope.go:122] "RemoveContainer" containerID="200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4" Oct 29 22:06:47.231154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.231086 4543 scope.go:122] "RemoveContainer" containerID="200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4" Oct 29 22:06:47.231641 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:47.231613 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4\": container with ID starting with 200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4 not found: ID does not exist" containerID="200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4" Oct 29 22:06:47.231741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.231652 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4"} err="failed to get container status \"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4\": rpc error: code = NotFound desc = could not find container \"200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4\": container with ID starting with 200e9ae20a85b3a5eacc4b9d500f7ae9f7681672f45a151a15d4a8819afec5a4 not found: ID does not exist" Oct 29 22:06:47.234569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.234545 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:06:47.236270 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:47.236247 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6010040_9244_4929_a78d_124820946161.slice/crio-d2d642cfa6f827b465faf6821e8bf2dd331b091184fb2d6e687692a6d6306bec WatchSource:0}: Error finding container d2d642cfa6f827b465faf6821e8bf2dd331b091184fb2d6e687692a6d6306bec: Status 404 returned error can't find the container with id d2d642cfa6f827b465faf6821e8bf2dd331b091184fb2d6e687692a6d6306bec Oct 29 22:06:47.282330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.282306 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:06:47.289155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.289134 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:06:47.290578 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:06:47.290553 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e38e3fb_56ca_4230_b182_5a8301badff4.slice/crio-eac4cf6f6db60410468bbc19bd0fab04b6fcf940947af4cd07aaab8bf0fa4150 WatchSource:0}: Error finding container eac4cf6f6db60410468bbc19bd0fab04b6fcf940947af4cd07aaab8bf0fa4150: Status 404 returned error can't find the container with id eac4cf6f6db60410468bbc19bd0fab04b6fcf940947af4cd07aaab8bf0fa4150 Oct 29 22:06:47.293324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:47.293300 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7046db-8577b44d67-xbt5d"] Oct 29 22:06:48.226455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.226414 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" event={"ID":"e6010040-9244-4929-a78d-124820946161","Type":"ContainerStarted","Data":"7611221da94f64f06b213cdac611c7c23d129aac6c6887d27e8c8ffe5085c7b5"} Oct 29 22:06:48.226900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.226505 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" event={"ID":"e6010040-9244-4929-a78d-124820946161","Type":"ContainerStarted","Data":"d2d642cfa6f827b465faf6821e8bf2dd331b091184fb2d6e687692a6d6306bec"} Oct 29 22:06:48.226900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.226667 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:06:48.228388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.228353 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" event={"ID":"1e38e3fb-56ca-4230-b182-5a8301badff4","Type":"ContainerStarted","Data":"069ae733561f5a8a651adf13dd0f3def42759fd4e6934e28cbf0cb4d4a0092c2"} Oct 29 22:06:48.228388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.228388 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" event={"ID":"1e38e3fb-56ca-4230-b182-5a8301badff4","Type":"ContainerStarted","Data":"eac4cf6f6db60410468bbc19bd0fab04b6fcf940947af4cd07aaab8bf0fa4150"} Oct 29 22:06:48.228555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.228484 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:48.271991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.271949 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" podStartSLOduration=2.271934488 podStartE2EDuration="2.271934488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:48.270829098 +0000 UTC m=+1470.012759772" watchObservedRunningTime="2026-10-29 22:06:48.271934488 +0000 UTC m=+1470.013865165" Oct 29 22:06:48.318889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.318847 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" podStartSLOduration=2.318834897 podStartE2EDuration="2.318834897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:48.315058509 +0000 UTC m=+1470.056989197" watchObservedRunningTime="2026-10-29 22:06:48.318834897 +0000 UTC m=+1470.060765569" Oct 29 22:06:48.387750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:48.387714 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" path="/var/lib/kubelet/pods/0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d/volumes" Oct 29 22:06:51.534666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.534628 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" podUID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" containerName="payload-processing" containerID="cri-o://ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c" gracePeriod=25 Oct 29 22:06:51.561909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.561879 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" podUID="c76b4bf9-2c4b-4030-b87c-11335fe04532" containerName="payload-pre-processing" containerID="cri-o://a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84" gracePeriod=25 Oct 29 22:06:51.837409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.837382 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:06:51.889993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.889970 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:06:51.969537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.969483 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\" (UniqueName: \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\") pod \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " Oct 29 22:06:51.969537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.969531 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\") pod \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\" (UID: \"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4\") " Oct 29 22:06:51.970069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.969986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume" pod "47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" (UID: "47d66cb9-ef44-490f-a9f7-0bbe7bee68f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:51.970239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.970222 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:51.971918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:51.971899 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb" pod "47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" (UID: "47d66cb9-ef44-490f-a9f7-0bbe7bee68f4"). InnerVolumeSpecName "kube-api-access-r76vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:52.070885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.070826 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\") pod \"c76b4bf9-2c4b-4030-b87c-11335fe04532\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " Oct 29 22:06:52.070987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.070906 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\" (UniqueName: \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\") pod \"c76b4bf9-2c4b-4030-b87c-11335fe04532\" (UID: \"c76b4bf9-2c4b-4030-b87c-11335fe04532\") " Oct 29 22:06:52.071266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.071244 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume" pod "c76b4bf9-2c4b-4030-b87c-11335fe04532" (UID: "c76b4bf9-2c4b-4030-b87c-11335fe04532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:52.071324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.071300 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r76vb\" (UniqueName: \"kubernetes.io/projected/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4-kube-api-access-r76vb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:52.073109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.073063 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4" pod "c76b4bf9-2c4b-4030-b87c-11335fe04532" (UID: "c76b4bf9-2c4b-4030-b87c-11335fe04532"). InnerVolumeSpecName "kube-api-access-r4rp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:52.172351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.172327 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76b4bf9-2c4b-4030-b87c-11335fe04532-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:52.172351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.172350 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rp4\" (UniqueName: \"kubernetes.io/projected/c76b4bf9-2c4b-4030-b87c-11335fe04532-kube-api-access-r4rp4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:06:52.259517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.259489 4543 generic.go:356] "Generic (PLEG): container finished" podID="c76b4bf9-2c4b-4030-b87c-11335fe04532" containerID="a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84" exitCode=0 Oct 29 22:06:52.259647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.259548 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" Oct 29 22:06:52.259647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.259568 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" event={"ID":"c76b4bf9-2c4b-4030-b87c-11335fe04532","Type":"ContainerDied","Data":"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84"} Oct 29 22:06:52.259647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.259646 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj" event={"ID":"c76b4bf9-2c4b-4030-b87c-11335fe04532","Type":"ContainerDied","Data":"f8608520e94c7c7ce65b7b73605436eef9256ecd8f7b91305bc6738847194ae5"} Oct 29 22:06:52.259825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.259668 4543 scope.go:122] "RemoveContainer" containerID="a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84" Oct 29 22:06:52.261275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.261251 4543 generic.go:356] "Generic (PLEG): container finished" podID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" containerID="ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c" exitCode=0 Oct 29 22:06:52.261397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.261286 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" event={"ID":"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4","Type":"ContainerDied","Data":"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c"} Oct 29 22:06:52.261397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.261334 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" event={"ID":"47d66cb9-ef44-490f-a9f7-0bbe7bee68f4","Type":"ContainerDied","Data":"fed7bc7d97bf6e362aed026a3dcf7eb14cbb36ffc96f11209237e9ed981eebd1"} Oct 29 22:06:52.261397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.261301 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77" Oct 29 22:06:52.271884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.271864 4543 scope.go:122] "RemoveContainer" containerID="a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84" Oct 29 22:06:52.272167 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:52.272147 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84\": container with ID starting with a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84 not found: ID does not exist" containerID="a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84" Oct 29 22:06:52.272241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.272172 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84"} err="failed to get container status \"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84\": rpc error: code = NotFound desc = could not find container \"a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84\": container with ID starting with a63965952961c0696845751d694a85ac38f1bbf32faa3fdb6a801e0f88576a84 not found: ID does not exist" Oct 29 22:06:52.272241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.272188 4543 scope.go:122] "RemoveContainer" containerID="ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c" Oct 29 22:06:52.282807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.282790 4543 scope.go:122] "RemoveContainer" containerID="ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c" Oct 29 22:06:52.283067 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:52.283050 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c\": container with ID starting with ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c not found: ID does not exist" containerID="ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c" Oct 29 22:06:52.283143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.283091 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c"} err="failed to get container status \"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c\": rpc error: code = NotFound desc = could not find container \"ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c\": container with ID starting with ddbbd86d7412427a77fec8277b0ced521f69efcd0decfeebfbf649fba834597c not found: ID does not exist" Oct 29 22:06:52.321149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.321061 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:06:52.332614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.332593 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-687f4c8855-4v7qj"] Oct 29 22:06:52.370851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.370827 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:06:52.376370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.376352 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-58fd766b8-zpc77"] Oct 29 22:06:52.386444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.386422 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" path="/var/lib/kubelet/pods/47d66cb9-ef44-490f-a9f7-0bbe7bee68f4/volumes" Oct 29 22:06:52.386836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:52.386821 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76b4bf9-2c4b-4030-b87c-11335fe04532" path="/var/lib/kubelet/pods/c76b4bf9-2c4b-4030-b87c-11335fe04532/volumes" Oct 29 22:06:57.961771 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:06:57.961729 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:06:59.238760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:59.238730 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:06:59.239275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:06:59.239139 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:07:01.213427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.213392 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:07:01.214159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214136 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c76b4bf9-2c4b-4030-b87c-11335fe04532" containerName="payload-pre-processing" Oct 29 22:07:01.214159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214157 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76b4bf9-2c4b-4030-b87c-11335fe04532" containerName="payload-pre-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214177 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" containerName="payload-pre-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214182 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" containerName="payload-pre-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214204 4543 cpu_manager.go:414] "containerMap: removing container" podUID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" containerName="payload-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214208 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" containerName="payload-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214333 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76b4bf9-2c4b-4030-b87c-11335fe04532" containerName="payload-pre-processing" Oct 29 22:07:01.214340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214344 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1fba0d-78f6-4d4a-b0c5-57011b3b7c2d" containerName="payload-pre-processing" Oct 29 22:07:01.214670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.214353 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d66cb9-ef44-490f-a9f7-0bbe7bee68f4" containerName="payload-processing" Oct 29 22:07:01.224458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.224436 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.239065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.239040 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:07:01.271603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.271578 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9d9498-openshift-default-dockercfg-wlzhv\"" Oct 29 22:07:01.276250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276225 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276261 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276387 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276523 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276556 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276664 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276834 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlskn\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276949 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.276979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.276976 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.377840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.377808 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.377869 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlskn\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.377909 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.377926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.377958 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378192 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378262 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378546 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378646 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378764 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.378975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.378954 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.380569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.380547 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.380768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.380747 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.403658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.403630 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.403958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.403936 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlskn\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\") pod \"e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.548160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.547347 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:01.918627 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:01.918598 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd1b6f9_c443_4c8f_8498_febb7813179c.slice/crio-249714c4577f02dea616e2e87d8e5d687141d1fdef59635084df8e04cc6ffe52 WatchSource:0}: Error finding container 249714c4577f02dea616e2e87d8e5d687141d1fdef59635084df8e04cc6ffe52: Status 404 returned error can't find the container with id 249714c4577f02dea616e2e87d8e5d687141d1fdef59635084df8e04cc6ffe52 Oct 29 22:07:01.919446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.919413 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:07:01.921946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.921908 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:01.922093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.921977 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:01.922093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:01.922019 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:02.374591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.374549 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerStarted","Data":"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7"} Oct 29 22:07:02.375112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.374604 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerStarted","Data":"249714c4577f02dea616e2e87d8e5d687141d1fdef59635084df8e04cc6ffe52"} Oct 29 22:07:02.455141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.453475 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podStartSLOduration=1.453453497 podStartE2EDuration="1.453453497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:02.4490636 +0000 UTC m=+1484.190994274" watchObservedRunningTime="2026-10-29 22:07:02.453453497 +0000 UTC m=+1484.195384172" Oct 29 22:07:02.503199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.503166 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:07:02.549228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.549186 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:02.559542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:02.558355 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:07:03.088232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.088200 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:07:03.092547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.092520 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.117093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.115672 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c53cc8-openshift-default-dockercfg-bq8qx\"" Oct 29 22:07:03.132100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.129838 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234423 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234472 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234546 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234597 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234669 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234755 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrt58\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.244101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.234876 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.335781 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.335879 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.335919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.335983 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336107 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336162 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrt58\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336231 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336270 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336705 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.336969 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.338241 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.338911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.338864 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.343099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.339146 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.345305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.343949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.345305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.344461 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.362682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.362661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrt58\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.364455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.364436 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\") pod \"e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.415888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.415862 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:03.614460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.597291 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:03.865372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.865335 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:07:03.867052 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:03.867026 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57b573_9951_4cb4_b668_a352bcbf0936.slice/crio-43e057b601df2521a9b867a0288a40717815f3169c328be2a0fd8785d04d618e WatchSource:0}: Error finding container 43e057b601df2521a9b867a0288a40717815f3169c328be2a0fd8785d04d618e: Status 404 returned error can't find the container with id 43e057b601df2521a9b867a0288a40717815f3169c328be2a0fd8785d04d618e Oct 29 22:07:03.870339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.870301 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:03.870451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.870371 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:03.870451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:03.870401 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:04.403101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.400801 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerStarted","Data":"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434"} Oct 29 22:07:04.403101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.400882 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerStarted","Data":"43e057b601df2521a9b867a0288a40717815f3169c328be2a0fd8785d04d618e"} Oct 29 22:07:04.403101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.400936 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:04.403101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.402104 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:07:04.417159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.416503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:04.425117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.423060 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:07:04.425117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.423127 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:07:04.463877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:04.463813 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podStartSLOduration=1.463795829 podStartE2EDuration="1.463795829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:04.460654509 +0000 UTC m=+1486.202585206" watchObservedRunningTime="2026-10-29 22:07:04.463795829 +0000 UTC m=+1486.205726505" Oct 29 22:07:05.420949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:05.420921 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:05.896796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:05.896763 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj_1e38e3fb-56ca-4230-b182-5a8301badff4/payload-processing/0.log" Oct 29 22:07:06.420174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.420143 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:06.420769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.420746 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:07:06.657054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.657017 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94"] Oct 29 22:07:06.661870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.661848 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.671817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.671794 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94"] Oct 29 22:07:06.679268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.679246 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:07:06.679494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.679479 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:07:06.692671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.692655 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:07:06.692901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.692887 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:07:06.724666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.724646 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.724852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.724830 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8cx\" (UniqueName: \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.724984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.724969 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.826593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.826568 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.826746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.826688 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8cx\" (UniqueName: \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.826820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.826807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.827285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.827260 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.857106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.854526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.857106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.857087 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8cx\" (UniqueName: \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\") pod \"maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:06.978359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:06.978328 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:07.359125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.359067 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94"] Oct 29 22:07:07.359889 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:07.359860 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b05b010_a59c_4d62_a11d_f3c689456e0b.slice/crio-5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a WatchSource:0}: Error finding container 5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a: Status 404 returned error can't find the container with id 5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a Oct 29 22:07:07.451100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.450423 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" event={"ID":"7b05b010-a59c-4d62-a11d-f3c689456e0b","Type":"ContainerStarted","Data":"5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a"} Oct 29 22:07:07.503764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.503721 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" podUID="35433c15-b08f-492e-a168-43789c3be5af" containerName="payload-processing" containerID="cri-o://e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485" gracePeriod=25 Oct 29 22:07:07.519713 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:07.519686 4543 logging.go:55] [core] [Channel #4067 SubChannel #4068]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.182:50052", ServerName: "10.133.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.182:50052: connect: connection refused" Oct 29 22:07:07.562020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.561850 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" podUID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" containerName="payload-pre-processing" containerID="cri-o://920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651" gracePeriod=25 Oct 29 22:07:07.690504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.690475 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:07:07.694953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.694933 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.712790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.712749 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9d9498\"" Oct 29 22:07:07.715282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.715262 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:07:07.724123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.721945 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9d9498-dockercfg-zvmhw\"" Oct 29 22:07:07.772684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.772255 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:07:07.783255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.779935 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.797956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.797914 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:07:07.869646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.869613 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.869992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.869970 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6g7\" (UniqueName: \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.870351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.870334 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.870483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.870469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzcj\" (UniqueName: \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.975144 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.975201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzcj\" (UniqueName: \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.975328 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.975351 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6g7\" (UniqueName: \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.976404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:07.977658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.977615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:07.989372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:07.988951 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:07:08.002649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.002596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6g7\" (UniqueName: \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\") pod \"payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:08.004067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.004020 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzcj\" (UniqueName: \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\") pod \"payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:08.025798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.025483 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:08.073944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.073585 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:07:08.076921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.076828 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\" (UniqueName: \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\") pod \"35433c15-b08f-492e-a168-43789c3be5af\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " Oct 29 22:07:08.077051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.077036 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\") pod \"35433c15-b08f-492e-a168-43789c3be5af\" (UID: \"35433c15-b08f-492e-a168-43789c3be5af\") " Oct 29 22:07:08.077814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.077508 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume" pod "35433c15-b08f-492e-a168-43789c3be5af" (UID: "35433c15-b08f-492e-a168-43789c3be5af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:08.077814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.077777 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35433c15-b08f-492e-a168-43789c3be5af-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:08.084624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.084588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz" pod "35433c15-b08f-492e-a168-43789c3be5af" (UID: "35433c15-b08f-492e-a168-43789c3be5af"). InnerVolumeSpecName "kube-api-access-gjzdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:08.190122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.182750 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\" (UniqueName: \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\") pod \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " Oct 29 22:07:08.190122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.182950 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\") pod \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\" (UID: \"13a0609e-332f-477e-b8d3-e3c24a7cd3b5\") " Oct 29 22:07:08.191421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.190868 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzdz\" (UniqueName: \"kubernetes.io/projected/35433c15-b08f-492e-a168-43789c3be5af-kube-api-access-gjzdz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:08.192269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.192210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume" pod "13a0609e-332f-477e-b8d3-e3c24a7cd3b5" (UID: "13a0609e-332f-477e-b8d3-e3c24a7cd3b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:08.194216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.194188 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km" pod "13a0609e-332f-477e-b8d3-e3c24a7cd3b5" (UID: "13a0609e-332f-477e-b8d3-e3c24a7cd3b5"). InnerVolumeSpecName "kube-api-access-4k5km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:08.213877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.213664 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:08.292652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.292624 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k5km\" (UniqueName: \"kubernetes.io/projected/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-kube-api-access-4k5km\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:08.292723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.292657 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a0609e-332f-477e-b8d3-e3c24a7cd3b5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:08.414543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.414521 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:07:08.470833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.470801 4543 generic.go:356] "Generic (PLEG): container finished" podID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" containerID="920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651" exitCode=0 Oct 29 22:07:08.470969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.470851 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" Oct 29 22:07:08.470969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.470879 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" event={"ID":"13a0609e-332f-477e-b8d3-e3c24a7cd3b5","Type":"ContainerDied","Data":"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651"} Oct 29 22:07:08.470969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.470934 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8" event={"ID":"13a0609e-332f-477e-b8d3-e3c24a7cd3b5","Type":"ContainerDied","Data":"ee4c4573f3886f982282670390982f08a36bd59c46133ac93d7d6a2fcca0a849"} Oct 29 22:07:08.470969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.470957 4543 scope.go:122] "RemoveContainer" containerID="920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651" Oct 29 22:07:08.472523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.472470 4543 generic.go:356] "Generic (PLEG): container finished" podID="35433c15-b08f-492e-a168-43789c3be5af" containerID="e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485" exitCode=0 Oct 29 22:07:08.472523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.472512 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" Oct 29 22:07:08.472523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.472546 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" event={"ID":"35433c15-b08f-492e-a168-43789c3be5af","Type":"ContainerDied","Data":"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485"} Oct 29 22:07:08.472523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.472591 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" event={"ID":"35433c15-b08f-492e-a168-43789c3be5af","Type":"ContainerDied","Data":"fab2d8f5702c082f6a3a2f5417514ca87c2af7a34d64706202be5266e90bc7be"} Oct 29 22:07:08.474369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.474349 4543 generic.go:356] "Generic (PLEG): container finished" podID="7b05b010-a59c-4d62-a11d-f3c689456e0b" containerID="66510eee1c3a8b0bdd5893e968b53ee7df207b103c66605770a6b202b45ac972" exitCode=0 Oct 29 22:07:08.474493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.474410 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" event={"ID":"7b05b010-a59c-4d62-a11d-f3c689456e0b","Type":"ContainerDied","Data":"66510eee1c3a8b0bdd5893e968b53ee7df207b103c66605770a6b202b45ac972"} Oct 29 22:07:08.475885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.475865 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" event={"ID":"39c04a79-34d1-4e9f-8d6e-059447089222","Type":"ContainerStarted","Data":"c29143af2fb4ee107e6ec80c68bfa5c8a1110defb9a533da98506e272eea4dc2"} Oct 29 22:07:08.488986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.488969 4543 scope.go:122] "RemoveContainer" containerID="920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651" Oct 29 22:07:08.489258 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:07:08.489235 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651\": container with ID starting with 920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651 not found: ID does not exist" containerID="920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651" Oct 29 22:07:08.489326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.489264 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651"} err="failed to get container status \"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651\": rpc error: code = NotFound desc = could not find container \"920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651\": container with ID starting with 920cdd9afd7633e593122580e03a097c05df4e2d4bcd2a7a5a1a640e70241651 not found: ID does not exist" Oct 29 22:07:08.489326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.489290 4543 scope.go:122] "RemoveContainer" containerID="e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485" Oct 29 22:07:08.514914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.514888 4543 scope.go:122] "RemoveContainer" containerID="e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485" Oct 29 22:07:08.515257 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:07:08.515233 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485\": container with ID starting with e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485 not found: ID does not exist" containerID="e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485" Oct 29 22:07:08.515348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.515277 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485"} err="failed to get container status \"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485\": rpc error: code = NotFound desc = could not find container \"e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485\": container with ID starting with e67df19d480ac1ee78ecd5fe7a9ca54148298fdda5f261dfa42aabf7ea5c9485 not found: ID does not exist" Oct 29 22:07:08.519598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.519554 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj" podUID="35433c15-b08f-492e-a168-43789c3be5af" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.182:50052\" within 1s: context deadline exceeded" Oct 29 22:07:08.587101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.572061 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:07:08.604880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.604790 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:07:08.622813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.622754 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-69cff988d7-4msz8"] Oct 29 22:07:08.669316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.669281 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:07:08.677317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:08.677271 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-8b9574ddd-8lkrj"] Oct 29 22:07:09.380936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.380902 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh"] Oct 29 22:07:09.386905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.386873 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7046db-5lvxh"] Oct 29 22:07:09.488178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.488133 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" event={"ID":"39c04a79-34d1-4e9f-8d6e-059447089222","Type":"ContainerStarted","Data":"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c"} Oct 29 22:07:09.488378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.488260 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:09.490028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.490000 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" event={"ID":"1ec94360-4262-4f8e-996b-6019f6747a6e","Type":"ContainerStarted","Data":"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07"} Oct 29 22:07:09.490204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.490046 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" event={"ID":"1ec94360-4262-4f8e-996b-6019f6747a6e","Type":"ContainerStarted","Data":"774bdca1e2916a5110657f23f2e6c45da0b032a8130ff704ec47b463396758d1"} Oct 29 22:07:09.490204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.490092 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:09.534977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.534865 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" podStartSLOduration=2.534846107 podStartE2EDuration="2.534846107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:09.532237267 +0000 UTC m=+1491.274167967" watchObservedRunningTime="2026-10-29 22:07:09.534846107 +0000 UTC m=+1491.276776813" Oct 29 22:07:09.598145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.598044 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" podStartSLOduration=2.598021561 podStartE2EDuration="2.598021561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:09.576998013 +0000 UTC m=+1491.318928687" watchObservedRunningTime="2026-10-29 22:07:09.598021561 +0000 UTC m=+1491.339952235" Oct 29 22:07:09.684448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.684415 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:09.822265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.822230 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\") pod \"7b05b010-a59c-4d62-a11d-f3c689456e0b\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " Oct 29 22:07:09.822462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.822339 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\") pod \"7b05b010-a59c-4d62-a11d-f3c689456e0b\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " Oct 29 22:07:09.822462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.822418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\" (UniqueName: \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\") pod \"7b05b010-a59c-4d62-a11d-f3c689456e0b\" (UID: \"7b05b010-a59c-4d62-a11d-f3c689456e0b\") " Oct 29 22:07:09.822664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.822635 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca" pod "7b05b010-a59c-4d62-a11d-f3c689456e0b" (UID: "7b05b010-a59c-4d62-a11d-f3c689456e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:09.823144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.823116 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.825522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.825474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca" pod "7b05b010-a59c-4d62-a11d-f3c689456e0b" (UID: "7b05b010-a59c-4d62-a11d-f3c689456e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:09.825851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.825793 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx" pod "7b05b010-a59c-4d62-a11d-f3c689456e0b" (UID: "7b05b010-a59c-4d62-a11d-f3c689456e0b"). InnerVolumeSpecName "kube-api-access-ml8cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:09.924753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.924679 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8cx\" (UniqueName: \"kubernetes.io/projected/7b05b010-a59c-4d62-a11d-f3c689456e0b-kube-api-access-ml8cx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:09.924753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:09.924713 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b05b010-a59c-4d62-a11d-f3c689456e0b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:10.387764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.387727 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" path="/var/lib/kubelet/pods/13a0609e-332f-477e-b8d3-e3c24a7cd3b5/volumes" Oct 29 22:07:10.388262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.388189 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35433c15-b08f-492e-a168-43789c3be5af" path="/var/lib/kubelet/pods/35433c15-b08f-492e-a168-43789c3be5af/volumes" Oct 29 22:07:10.388603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.388584 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af25e76b-2707-415f-9430-0830c7b22ba8" path="/var/lib/kubelet/pods/af25e76b-2707-415f-9430-0830c7b22ba8/volumes" Oct 29 22:07:10.454915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.454890 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:07:10.455753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455732 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7b05b010-a59c-4d62-a11d-f3c689456e0b" containerName="revoke-keys" Oct 29 22:07:10.455753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455753 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b05b010-a59c-4d62-a11d-f3c689456e0b" containerName="revoke-keys" Oct 29 22:07:10.455881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455768 4543 cpu_manager.go:414] "containerMap: removing container" podUID="35433c15-b08f-492e-a168-43789c3be5af" containerName="payload-processing" Oct 29 22:07:10.455881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455773 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35433c15-b08f-492e-a168-43789c3be5af" containerName="payload-processing" Oct 29 22:07:10.455881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455797 4543 cpu_manager.go:414] "containerMap: removing container" podUID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" containerName="payload-pre-processing" Oct 29 22:07:10.455881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455802 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" containerName="payload-pre-processing" Oct 29 22:07:10.456003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455890 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a0609e-332f-477e-b8d3-e3c24a7cd3b5" containerName="payload-pre-processing" Oct 29 22:07:10.456003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455905 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b05b010-a59c-4d62-a11d-f3c689456e0b" containerName="revoke-keys" Oct 29 22:07:10.456003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.455912 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="35433c15-b08f-492e-a168-43789c3be5af" containerName="payload-processing" Oct 29 22:07:10.461023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.461005 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.473299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.473273 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:07:10.478109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.478057 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.479785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.479763 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:07:10.484395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.484372 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c53cc8\"" Oct 29 22:07:10.484510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.484489 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c53cc8-dockercfg-gt9dx\"" Oct 29 22:07:10.523197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.499208 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:07:10.523197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.521567 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" Oct 29 22:07:10.523197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.521882 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94" event={"ID":"7b05b010-a59c-4d62-a11d-f3c689456e0b","Type":"ContainerDied","Data":"5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a"} Oct 29 22:07:10.523197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.521934 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5605dd3836cc3bd1eac2064ffbe94cce986301c827c6a8436c8c5ceebf35dc5a" Oct 29 22:07:10.532700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.532666 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmxm\" (UniqueName: \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.532879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.532761 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smgn\" (UniqueName: \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.532879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.532869 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.533279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.533239 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.635154 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.635257 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmxm\" (UniqueName: \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.635333 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smgn\" (UniqueName: \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.635422 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.636311 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.649363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.636960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.665332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.665302 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmxm\" (UniqueName: \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\") pod \"payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:10.665945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.665916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smgn\" (UniqueName: \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\") pod \"payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.782546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.782512 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:10.797883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:10.797853 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:11.103412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.103383 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:07:11.104415 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:11.104391 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d7773e_afe5_4e37_a01b_41043a5a4886.slice/crio-a74130e3efaaa7b401834806c01503d9e424212dcd6333305ca5a39717ac9827 WatchSource:0}: Error finding container a74130e3efaaa7b401834806c01503d9e424212dcd6333305ca5a39717ac9827: Status 404 returned error can't find the container with id a74130e3efaaa7b401834806c01503d9e424212dcd6333305ca5a39717ac9827 Oct 29 22:07:11.154413 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:11.154380 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d774a86_2104_4b4a_a0ee_366322d1bfb4.slice/crio-11bd3d5ef22f96c672c9ce4a5c035864b2f3480eaa08ae47f55819e4c7ba397e WatchSource:0}: Error finding container 11bd3d5ef22f96c672c9ce4a5c035864b2f3480eaa08ae47f55819e4c7ba397e: Status 404 returned error can't find the container with id 11bd3d5ef22f96c672c9ce4a5c035864b2f3480eaa08ae47f55819e4c7ba397e Oct 29 22:07:11.154858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.154837 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:07:11.531663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.531632 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" event={"ID":"23d7773e-afe5-4e37-a01b-41043a5a4886","Type":"ContainerStarted","Data":"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a"} Oct 29 22:07:11.532180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.531686 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" event={"ID":"23d7773e-afe5-4e37-a01b-41043a5a4886","Type":"ContainerStarted","Data":"a74130e3efaaa7b401834806c01503d9e424212dcd6333305ca5a39717ac9827"} Oct 29 22:07:11.532180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.531728 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:11.533861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.533838 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" event={"ID":"3d774a86-2104-4b4a-a0ee-366322d1bfb4","Type":"ContainerStarted","Data":"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb"} Oct 29 22:07:11.533973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.533902 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:11.533973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.533929 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" event={"ID":"3d774a86-2104-4b4a-a0ee-366322d1bfb4","Type":"ContainerStarted","Data":"11bd3d5ef22f96c672c9ce4a5c035864b2f3480eaa08ae47f55819e4c7ba397e"} Oct 29 22:07:11.577597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.577547 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" podStartSLOduration=1.57753397 podStartE2EDuration="1.57753397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:11.575284077 +0000 UTC m=+1493.317214754" watchObservedRunningTime="2026-10-29 22:07:11.57753397 +0000 UTC m=+1493.319464643" Oct 29 22:07:11.623813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:11.623771 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" podStartSLOduration=1.6237586259999999 podStartE2EDuration="1.623758626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:11.620294555 +0000 UTC m=+1493.362225228" watchObservedRunningTime="2026-10-29 22:07:11.623758626 +0000 UTC m=+1493.365689298" Oct 29 22:07:12.543184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.543150 4543 generic.go:356] "Generic (PLEG): container finished" podID="9dea599f-9164-4a38-b80d-0de739784091" containerID="ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0" exitCode=0 Oct 29 22:07:12.543696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.543225 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerDied","Data":"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0"} Oct 29 22:07:12.543831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.543814 4543 scope.go:122] "RemoveContainer" containerID="ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0" Oct 29 22:07:12.545250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.545219 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:12.545367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.545304 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:12.545367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:12.545355 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:13.558490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:13.558449 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerStarted","Data":"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1"} Oct 29 22:07:13.766065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:13.766026 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:07:13.767472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:13.767441 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:07:13.767619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:13.767498 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:07:14.765390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:14.765351 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:07:14.765769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:14.765408 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:07:15.294728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:15.294694 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:07:15.765841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:15.765803 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:07:15.766294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:15.765858 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:07:16.765730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:16.765696 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:07:16.765901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:16.765748 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:07:17.518446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:17.518405 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:07:17.518662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:17.518636 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" containerID="cri-o://caff5cff29bcb76ca2b3786267cf0ba0eac0618b2cf99bbc77486950207fa38e" gracePeriod=30 Oct 29 22:07:17.768605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:17.768500 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 29 22:07:17.768605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:17.768575 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 29 22:07:18.844493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:18.844463 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:07:18.845969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:18.845929 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:07:19.097592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.097499 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:07:19.102559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.102531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.124092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.123136 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9b0939-openshift-default-dockercfg-7cxqn\"" Oct 29 22:07:19.137129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.129640 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:07:19.230620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230577 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.230822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.230822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230796 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.230946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230851 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.230946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230893 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4r8\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.231056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.230963 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.231150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.231070 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.231150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.231133 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.231261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.231207 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332046 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332173 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332283 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332380 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332440 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332474 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4r8\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332518 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332527 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332635 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.332743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.332718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.333060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.333033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.333060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.333048 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.334713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.334694 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.335194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.335173 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.362052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.361036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4r8\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.362776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.362726 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\") pod \"e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.430708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.430669 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:19.763673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.763653 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:07:19.765190 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:19.765166 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d6b37a_36f0_4d25_8c5e_0f7dd13a99f5.slice/crio-661ec643f5dc6fd49d35e71ca092790d218f3d9faa173f2263eeaf1271af0f53 WatchSource:0}: Error finding container 661ec643f5dc6fd49d35e71ca092790d218f3d9faa173f2263eeaf1271af0f53: Status 404 returned error can't find the container with id 661ec643f5dc6fd49d35e71ca092790d218f3d9faa173f2263eeaf1271af0f53 Oct 29 22:07:19.767754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.767715 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:19.767846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.767800 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:19.767918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:19.767847 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:20.192201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.192118 4543 scope.go:122] "RemoveContainer" containerID="ea295ba511d74828e60e93d5de36f9dec398b4d90251145259c5ab8ee22e76e6" Oct 29 22:07:20.203419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.203385 4543 scope.go:122] "RemoveContainer" containerID="73cb0a890b936e86ed0a61dccb62ab79394bd4cd14441ce544719378ac1382d3" Oct 29 22:07:20.218201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.218178 4543 scope.go:122] "RemoveContainer" containerID="bb9c58eedc02af2d0883fa0ef11e48893b77dbd6830ad484bce82c5700cbb986" Oct 29 22:07:20.392876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.392845 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:07:20.398015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.397996 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.421037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.421008 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:07:20.433483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.433461 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a20be1\"" Oct 29 22:07:20.433629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.433563 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a20be1-dockercfg-74sfm\"" Oct 29 22:07:20.532129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.528758 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:07:20.532129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.530720 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:07:20.547976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.547945 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:07:20.552366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.552338 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hwd\" (UniqueName: \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.552572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.552494 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.552649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.552633 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.588004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.587973 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:07:20.657678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.657637 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.657974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.657955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.658059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.658025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hwd\" (UniqueName: \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.659008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.658057 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9h7\" (UniqueName: \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.659371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.659347 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.662639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.662583 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerStarted","Data":"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b"} Oct 29 22:07:20.662778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.662650 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerStarted","Data":"661ec643f5dc6fd49d35e71ca092790d218f3d9faa173f2263eeaf1271af0f53"} Oct 29 22:07:20.686998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.686959 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hwd\" (UniqueName: \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\") pod \"payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.729283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.729240 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:20.729481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.729223 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podStartSLOduration=1.729207084 podStartE2EDuration="1.729207084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:20.719798467 +0000 UTC m=+1502.461729140" watchObservedRunningTime="2026-10-29 22:07:20.729207084 +0000 UTC m=+1502.471137756" Oct 29 22:07:20.764768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.760546 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.764768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.760629 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9h7\" (UniqueName: \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.764768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.761789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.792989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.792882 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9h7\" (UniqueName: \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\") pod \"payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:20.889741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:20.889702 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:21.075142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.075059 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:07:21.075807 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:21.075778 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32046f70_e24b_4ff2_96d8_b5e03b08d5ac.slice/crio-ae35a1688a00a33344a254f0c644fdd81f1bf4babd951892bf0e2ed0bba1ae2d WatchSource:0}: Error finding container ae35a1688a00a33344a254f0c644fdd81f1bf4babd951892bf0e2ed0bba1ae2d: Status 404 returned error can't find the container with id ae35a1688a00a33344a254f0c644fdd81f1bf4babd951892bf0e2ed0bba1ae2d Oct 29 22:07:21.209638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.209615 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:07:21.211038 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:21.211006 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed3194e_7db5_40f4_8b9e_d925e7f051ab.slice/crio-4331f2b02e218eee6f5eaf007ef57181aa48d2178658a8a96320797e4b0275d6 WatchSource:0}: Error finding container 4331f2b02e218eee6f5eaf007ef57181aa48d2178658a8a96320797e4b0275d6: Status 404 returned error can't find the container with id 4331f2b02e218eee6f5eaf007ef57181aa48d2178658a8a96320797e4b0275d6 Oct 29 22:07:21.431458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.431367 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:21.438218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.438193 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:21.675693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.675651 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" event={"ID":"32046f70-e24b-4ff2-96d8-b5e03b08d5ac","Type":"ContainerStarted","Data":"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f"} Oct 29 22:07:21.675879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.675720 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" event={"ID":"32046f70-e24b-4ff2-96d8-b5e03b08d5ac","Type":"ContainerStarted","Data":"ae35a1688a00a33344a254f0c644fdd81f1bf4babd951892bf0e2ed0bba1ae2d"} Oct 29 22:07:21.675879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.675866 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:21.677644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.677618 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" event={"ID":"0ed3194e-7db5-40f4-8b9e-d925e7f051ab","Type":"ContainerStarted","Data":"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6"} Oct 29 22:07:21.677771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.677670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" event={"ID":"0ed3194e-7db5-40f4-8b9e-d925e7f051ab","Type":"ContainerStarted","Data":"4331f2b02e218eee6f5eaf007ef57181aa48d2178658a8a96320797e4b0275d6"} Oct 29 22:07:21.677771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.677725 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:21.677771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.677768 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:21.678855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.678836 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:07:21.720688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.720640 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" podStartSLOduration=1.720624966 podStartE2EDuration="1.720624966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:21.719470812 +0000 UTC m=+1503.461401488" watchObservedRunningTime="2026-10-29 22:07:21.720624966 +0000 UTC m=+1503.462555641" Oct 29 22:07:21.818336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:21.818285 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" podStartSLOduration=1.818272378 podStartE2EDuration="1.818272378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:21.816327495 +0000 UTC m=+1503.558258180" watchObservedRunningTime="2026-10-29 22:07:21.818272378 +0000 UTC m=+1503.560203090" Oct 29 22:07:22.546574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.546543 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:07:22.547062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.546612 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:07:22.611151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.611111 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 29 22:07:22.611273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.611174 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 29 22:07:22.691053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.691024 4543 generic.go:356] "Generic (PLEG): container finished" podID="21dce852-8447-4004-989d-86ee0f6433a3" containerID="caff5cff29bcb76ca2b3786267cf0ba0eac0618b2cf99bbc77486950207fa38e" exitCode=0 Oct 29 22:07:22.691472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.691142 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerDied","Data":"caff5cff29bcb76ca2b3786267cf0ba0eac0618b2cf99bbc77486950207fa38e"} Oct 29 22:07:22.691472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.691231 4543 scope.go:122] "RemoveContainer" containerID="e75f2de3effe67c49540147ef2d166ddae21ebe2385e749e8327198ba4e33a04" Oct 29 22:07:22.859511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.859489 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:07:22.995339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995309 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995342 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995368 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995451 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995493 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995564 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995622 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995650 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995692 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\") pod \"21dce852-8447-4004-989d-86ee0f6433a3\" (UID: \"21dce852-8447-4004-989d-86ee0f6433a3\") " Oct 29 22:07:22.995797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995706 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:22.995797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:22.995984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.995962 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:22.996039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996004 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:22.996236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:22.996450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996429 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:22.996450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996452 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:22.996790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996462 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:22.996790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996470 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce852-8447-4004-989d-86ee0f6433a3-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:22.996790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.996481 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:22.998273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.998251 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "kube-api-access-mscth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:22.998362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.998317 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:07:22.998428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.998371 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:22.998468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:22.998432 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token" pod "21dce852-8447-4004-989d-86ee0f6433a3" (UID: "21dce852-8447-4004-989d-86ee0f6433a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:23.098197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.098141 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce852-8447-4004-989d-86ee0f6433a3-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:23.098197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.098162 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mscth\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-kube-api-access-mscth\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:23.098197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.098172 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce852-8447-4004-989d-86ee0f6433a3-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:23.098197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.098181 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce852-8447-4004-989d-86ee0f6433a3-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:23.664591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.664561 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w"] Oct 29 22:07:23.665373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.665356 4543 cpu_manager.go:414] "containerMap: removing container" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:23.665438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.665376 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:23.665514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.665505 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:23.665554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.665519 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:23.668254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.668237 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.685565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.685546 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:07:23.698292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.698266 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:07:23.698433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.698335 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:07:23.698498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.698442 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:07:23.703035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.703015 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" Oct 29 22:07:23.703198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.703057 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6" event={"ID":"21dce852-8447-4004-989d-86ee0f6433a3","Type":"ContainerDied","Data":"c820b16a2e346a1cf1281e9bc3ae2b4081d91a1235d9269e8d0b036016f39ad1"} Oct 29 22:07:23.703198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.703170 4543 scope.go:122] "RemoveContainer" containerID="caff5cff29bcb76ca2b3786267cf0ba0eac0618b2cf99bbc77486950207fa38e" Oct 29 22:07:23.703394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.703374 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w"] Oct 29 22:07:23.761421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.761399 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:07:23.767287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.767260 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7046db-openshift-default-866fbf48b9-4bzm6"] Oct 29 22:07:23.806446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.806417 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.806734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.806712 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.806979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.806955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.908046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.908016 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.908263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.908243 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.908350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.908304 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.908989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.908960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.910763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.910739 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.934482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.934419 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") pod \"maas-api-revoke-sub-acb66b6c-8887w\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:23.982274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:23.982254 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:24.300487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:24.300460 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w"] Oct 29 22:07:24.302286 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:24.302256 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be832ca_8136_4efe_8cac_167b3298fbed.slice/crio-99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7 WatchSource:0}: Error finding container 99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7: Status 404 returned error can't find the container with id 99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7 Oct 29 22:07:24.386604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:24.386569 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dce852-8447-4004-989d-86ee0f6433a3" path="/var/lib/kubelet/pods/21dce852-8447-4004-989d-86ee0f6433a3/volumes" Oct 29 22:07:24.713265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:24.713228 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" event={"ID":"9be832ca-8136-4efe-8cac-167b3298fbed","Type":"ContainerStarted","Data":"99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7"} Oct 29 22:07:25.723674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:25.723644 4543 generic.go:356] "Generic (PLEG): container finished" podID="9be832ca-8136-4efe-8cac-167b3298fbed" containerID="275718283c32612f44a226f2e4e10a92c05530d904379595ed70cf62f94de545" exitCode=0 Oct 29 22:07:25.724125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:25.723687 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" event={"ID":"9be832ca-8136-4efe-8cac-167b3298fbed","Type":"ContainerDied","Data":"275718283c32612f44a226f2e4e10a92c05530d904379595ed70cf62f94de545"} Oct 29 22:07:26.865349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.865317 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:07:26.866137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.866114 4543 cpu_manager.go:414] "containerMap: removing container" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:26.866137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.866136 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dce852-8447-4004-989d-86ee0f6433a3" containerName="istio-proxy" Oct 29 22:07:26.869218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.869193 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:26.891564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.891354 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:07:26.899689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.899667 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9b0939\"" Oct 29 22:07:26.899787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.899770 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9b0939-dockercfg-njzxx\"" Oct 29 22:07:26.906551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.906526 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:26.986182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.983130 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:07:26.986182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.984229 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9be832ca-8136-4efe-8cac-167b3298fbed" containerName="revoke-keys" Oct 29 22:07:26.986182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.984249 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be832ca-8136-4efe-8cac-167b3298fbed" containerName="revoke-keys" Oct 29 22:07:26.986182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:26.984451 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be832ca-8136-4efe-8cac-167b3298fbed" containerName="revoke-keys" Oct 29 22:07:27.016302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.001105 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\") pod \"9be832ca-8136-4efe-8cac-167b3298fbed\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " Oct 29 22:07:27.016302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.001447 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\") pod \"9be832ca-8136-4efe-8cac-167b3298fbed\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " Oct 29 22:07:27.016302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.001495 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") pod \"9be832ca-8136-4efe-8cac-167b3298fbed\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " Oct 29 22:07:27.016302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.001910 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk62\" (UniqueName: \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.016302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.002109 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.042098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.037859 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca" pod "9be832ca-8136-4efe-8cac-167b3298fbed" (UID: "9be832ca-8136-4efe-8cac-167b3298fbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:27.050187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.044410 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:07:27.050187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.044538 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.114142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.112339 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb" pod "9be832ca-8136-4efe-8cac-167b3298fbed" (UID: "9be832ca-8136-4efe-8cac-167b3298fbed"). InnerVolumeSpecName "kube-api-access-lnxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:27.131151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.114590 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") pod \"9be832ca-8136-4efe-8cac-167b3298fbed\" (UID: \"9be832ca-8136-4efe-8cac-167b3298fbed\") " Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.131788 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk62\" (UniqueName: \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.134237 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.134433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.134484 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5dm\" (UniqueName: \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.134627 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:27.116368 4543 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/9be832ca-8136-4efe-8cac-167b3298fbed/volumes/kubernetes.io~projected/kube-api-access-lnxxb Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.134728 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb" pod "9be832ca-8136-4efe-8cac-167b3298fbed" (UID: "9be832ca-8136-4efe-8cac-167b3298fbed"). InnerVolumeSpecName "kube-api-access-lnxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:27.137265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.135802 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.147203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.147171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca" pod "9be832ca-8136-4efe-8cac-167b3298fbed" (UID: "9be832ca-8136-4efe-8cac-167b3298fbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:27.164378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.163850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk62\" (UniqueName: \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\") pod \"payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.217045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.217014 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.236169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.236130 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.236283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.236211 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5dm\" (UniqueName: \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.236502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.236486 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxxb\" (UniqueName: \"kubernetes.io/projected/9be832ca-8136-4efe-8cac-167b3298fbed-kube-api-access-lnxxb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:27.236536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.236511 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be832ca-8136-4efe-8cac-167b3298fbed-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:27.236773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.236755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.262937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.262903 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5dm\" (UniqueName: \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\") pod \"payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.467511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.467484 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:27.529406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.528409 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:07:27.532888 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:27.532862 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d863c8d_b4e7_4fe3_a461_29e144aff444.slice/crio-9df19a8ad465199f43a9dc8c270b619a433c1bede77de31e7ab8372589e255e5 WatchSource:0}: Error finding container 9df19a8ad465199f43a9dc8c270b619a433c1bede77de31e7ab8372589e255e5: Status 404 returned error can't find the container with id 9df19a8ad465199f43a9dc8c270b619a433c1bede77de31e7ab8372589e255e5 Oct 29 22:07:27.742210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.742188 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" Oct 29 22:07:27.742339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.742227 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w" event={"ID":"9be832ca-8136-4efe-8cac-167b3298fbed","Type":"ContainerDied","Data":"99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7"} Oct 29 22:07:27.742339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.742283 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d6278897ca8d58eb4d06c0ac6fc7c63a308b5d3f3d70c84e347bd0af3ac8d7" Oct 29 22:07:27.744130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.744108 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" event={"ID":"9d863c8d-b4e7-4fe3-a461-29e144aff444","Type":"ContainerStarted","Data":"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5"} Oct 29 22:07:27.744227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.744149 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" event={"ID":"9d863c8d-b4e7-4fe3-a461-29e144aff444","Type":"ContainerStarted","Data":"9df19a8ad465199f43a9dc8c270b619a433c1bede77de31e7ab8372589e255e5"} Oct 29 22:07:27.744227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.744171 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:27.779428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.779404 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:07:27.780259 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:27.780235 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd11e93a_0e42_4362_b1f1_3284b29d1e17.slice/crio-4bd177e3b6d57f31bdc27dc55543c853dbb6bbed92cd71fe60e2220a08a992d7 WatchSource:0}: Error finding container 4bd177e3b6d57f31bdc27dc55543c853dbb6bbed92cd71fe60e2220a08a992d7: Status 404 returned error can't find the container with id 4bd177e3b6d57f31bdc27dc55543c853dbb6bbed92cd71fe60e2220a08a992d7 Oct 29 22:07:27.793098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:27.793037 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" podStartSLOduration=1.793022911 podStartE2EDuration="1.793022911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:27.787480843 +0000 UTC m=+1509.529411516" watchObservedRunningTime="2026-10-29 22:07:27.793022911 +0000 UTC m=+1509.534953586" Oct 29 22:07:27.961479 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:07:27.961443 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:07:28.041276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.041194 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4"] Oct 29 22:07:28.045360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.045329 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f00ced5-lkdn4"] Oct 29 22:07:28.387157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.387058 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429e313e-6058-4893-913a-d7483f70a346" path="/var/lib/kubelet/pods/429e313e-6058-4893-913a-d7483f70a346/volumes" Oct 29 22:07:28.753299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.753268 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" event={"ID":"bd11e93a-0e42-4362-b1f1-3284b29d1e17","Type":"ContainerStarted","Data":"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911"} Oct 29 22:07:28.753496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.753437 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:28.753496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.753458 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" event={"ID":"bd11e93a-0e42-4362-b1f1-3284b29d1e17","Type":"ContainerStarted","Data":"4bd177e3b6d57f31bdc27dc55543c853dbb6bbed92cd71fe60e2220a08a992d7"} Oct 29 22:07:28.798033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:28.797992 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" podStartSLOduration=2.797979235 podStartE2EDuration="2.797979235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:28.79615316 +0000 UTC m=+1510.538083834" watchObservedRunningTime="2026-10-29 22:07:28.797979235 +0000 UTC m=+1510.539909911" Oct 29 22:07:31.924171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:31.924139 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94"] Oct 29 22:07:31.966273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:31.966226 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-442dc2-jpv94"] Oct 29 22:07:32.387587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:32.387550 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b05b010-a59c-4d62-a11d-f3c689456e0b" path="/var/lib/kubelet/pods/7b05b010-a59c-4d62-a11d-f3c689456e0b/volumes" Oct 29 22:07:32.693862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:32.693784 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:07:32.694658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:32.694632 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:07:33.767102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.762891 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:07:33.797135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.797060 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:07:33.797452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.797435 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:33.820484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.820431 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fd1d4e\"" Oct 29 22:07:33.844805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.841885 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fd1d4e-dockercfg-k555s\"" Oct 29 22:07:33.908925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.908896 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:07:33.912971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.912950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:33.915762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.915739 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:07:33.992890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.992858 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xl4r\" (UniqueName: \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:33.993135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:33.993065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.094148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.094049 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.094148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.094131 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xl4r\" (UniqueName: \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.094334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.094282 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dkj\" (UniqueName: \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.094376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.094339 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.095043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.095022 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.120185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.120158 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xl4r\" (UniqueName: \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\") pod \"payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.195703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.195675 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dkj\" (UniqueName: \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.195872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.195859 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.196543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.196521 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.206349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.206326 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.221778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.221751 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dkj\" (UniqueName: \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\") pod \"payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.227470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.227450 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.551648 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:34.551618 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79034d70_eb2d_4f48_a4b2_d2b650ce6975.slice/crio-eede619fff9be6e966f64f83c6fc4612d945e9453d18619ba384ee748fa89108 WatchSource:0}: Error finding container eede619fff9be6e966f64f83c6fc4612d945e9453d18619ba384ee748fa89108: Status 404 returned error can't find the container with id eede619fff9be6e966f64f83c6fc4612d945e9453d18619ba384ee748fa89108 Oct 29 22:07:34.551648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.551623 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:07:34.591676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.591650 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:07:34.592509 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:34.592474 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3650003a_e544_4cf4_a880_7ff2b8849d55.slice/crio-14669e8189351d759dbd85cd533d584a1766919e1b056eba55fc1a90159574be WatchSource:0}: Error finding container 14669e8189351d759dbd85cd533d584a1766919e1b056eba55fc1a90159574be: Status 404 returned error can't find the container with id 14669e8189351d759dbd85cd533d584a1766919e1b056eba55fc1a90159574be Oct 29 22:07:34.894946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.894856 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" event={"ID":"3650003a-e544-4cf4-a880-7ff2b8849d55","Type":"ContainerStarted","Data":"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471"} Oct 29 22:07:34.894946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.894923 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" event={"ID":"3650003a-e544-4cf4-a880-7ff2b8849d55","Type":"ContainerStarted","Data":"14669e8189351d759dbd85cd533d584a1766919e1b056eba55fc1a90159574be"} Oct 29 22:07:34.895435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.894951 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:34.897299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.897270 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" event={"ID":"79034d70-eb2d-4f48-a4b2-d2b650ce6975","Type":"ContainerStarted","Data":"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7"} Oct 29 22:07:34.897418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.897318 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" event={"ID":"79034d70-eb2d-4f48-a4b2-d2b650ce6975","Type":"ContainerStarted","Data":"eede619fff9be6e966f64f83c6fc4612d945e9453d18619ba384ee748fa89108"} Oct 29 22:07:34.897418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.897355 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:34.942342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.942296 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" podStartSLOduration=1.942284135 podStartE2EDuration="1.942284135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:34.938812045 +0000 UTC m=+1516.680742717" watchObservedRunningTime="2026-10-29 22:07:34.942284135 +0000 UTC m=+1516.684214808" Oct 29 22:07:34.986239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:34.986199 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" podStartSLOduration=1.98618608 podStartE2EDuration="1.98618608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:34.983869594 +0000 UTC m=+1516.725800267" watchObservedRunningTime="2026-10-29 22:07:34.98618608 +0000 UTC m=+1516.728116806" Oct 29 22:07:38.758204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.758173 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:07:38.940487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.940457 4543 generic.go:356] "Generic (PLEG): container finished" podID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerID="a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b" exitCode=0 Oct 29 22:07:38.940670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.940533 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerDied","Data":"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b"} Oct 29 22:07:38.941037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.941022 4543 scope.go:122] "RemoveContainer" containerID="a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b" Oct 29 22:07:38.942982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.942947 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:38.943138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.943030 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:38.943138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:38.943101 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:07:39.773527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:39.773493 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:07:39.967605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:39.967573 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerStarted","Data":"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0"} Oct 29 22:07:40.144160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:40.141746 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:07:40.144160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:40.141870 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:40.144160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:40.141913 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:41.152911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.152876 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:41.153633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.153598 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:41.356102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.355189 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k"] Oct 29 22:07:41.359985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.359958 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.361811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.361787 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k"] Oct 29 22:07:41.377597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.377573 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:07:41.389976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.389954 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:07:41.414362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.414305 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:07:41.414494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.414403 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:07:41.453098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.452101 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hzz\" (UniqueName: \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.453098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.452285 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.453098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.452369 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.562157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.556230 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.562157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.556338 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.562157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.556490 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hzz\" (UniqueName: \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.565981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.565945 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.571165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.569503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.586234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.586208 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hzz\" (UniqueName: \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.675433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.675356 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:41.983038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:41.983007 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:07:42.004125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.004099 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k"] Oct 29 22:07:42.005358 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:42.005333 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7557f8b_d351_4d59_87ca_62e277350a0f.slice/crio-f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f WatchSource:0}: Error finding container f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f: Status 404 returned error can't find the container with id f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f Oct 29 22:07:42.024981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.024959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" event={"ID":"f7557f8b-d351-4d59-87ca-62e277350a0f","Type":"ContainerStarted","Data":"f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f"} Oct 29 22:07:42.143155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.139539 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:42.143155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.139592 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:42.289243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.289216 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t"] Oct 29 22:07:42.293638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.293616 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.313038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.313013 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t"] Oct 29 22:07:42.380669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.380637 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.381028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.381003 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.381107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.381088 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjs5j\" (UniqueName: \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.482386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.482341 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.482544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.482415 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjs5j\" (UniqueName: \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.482544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.482443 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.483176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.483147 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.485187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.485164 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.508872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.508805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjs5j\" (UniqueName: \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\") pod \"maas-api-revoke-sub-c4b475f6-qbt2t\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.606014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.605983 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:42.792270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.791533 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r"] Oct 29 22:07:42.798622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.798599 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.815004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.814976 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r"] Oct 29 22:07:42.890875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.890840 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.891058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.891023 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.891166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.891130 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7zf\" (UniqueName: \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.992739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.992711 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.992911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.992778 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7zf\" (UniqueName: \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.992911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.992901 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.993496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.993470 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:42.995359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:42.995333 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:43.015241 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:43.015219 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e155b22_7abe_42d4_9100_ee19e3083836.slice/crio-a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5 WatchSource:0}: Error finding container a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5: Status 404 returned error can't find the container with id a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5 Oct 29 22:07:43.016728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.016701 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t"] Oct 29 22:07:43.018748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.018727 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7zf\" (UniqueName: \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\") pod \"maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:43.059903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.059872 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" event={"ID":"3e155b22-7abe-42d4-9100-ee19e3083836","Type":"ContainerStarted","Data":"a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5"} Oct 29 22:07:43.063128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.063103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" event={"ID":"f7557f8b-d351-4d59-87ca-62e277350a0f","Type":"ContainerStarted","Data":"6f3fd21f1be66caa3f3be45f89a1b781fde71dd20501de456ace17a4c5139969"} Oct 29 22:07:43.111096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.110109 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" podStartSLOduration=2.110091571 podStartE2EDuration="2.110091571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:43.109657165 +0000 UTC m=+1524.851587839" watchObservedRunningTime="2026-10-29 22:07:43.110091571 +0000 UTC m=+1524.852022248" Oct 29 22:07:43.153101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.146163 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:43.153101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.146220 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:43.171408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.171366 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:43.569229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:43.569194 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r"] Oct 29 22:07:43.571402 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:07:43.571371 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode900d3c2_52a2_44af_bc16_534c94a35041.slice/crio-7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064 WatchSource:0}: Error finding container 7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064: Status 404 returned error can't find the container with id 7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064 Oct 29 22:07:44.072740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.072705 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" event={"ID":"e900d3c2-52a2-44af-bc16-534c94a35041","Type":"ContainerStarted","Data":"5f9be9b7cd3a5c8232b095c4f125a2944e23d58dde7cc4fae17f209830d0f9e4"} Oct 29 22:07:44.072887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.072767 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" event={"ID":"e900d3c2-52a2-44af-bc16-534c94a35041","Type":"ContainerStarted","Data":"7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064"} Oct 29 22:07:44.074647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.074619 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" event={"ID":"3e155b22-7abe-42d4-9100-ee19e3083836","Type":"ContainerStarted","Data":"c8f03a4a9736784e7ee8d759f2e463afea6b7f5c006489d779ffeb3760dd088c"} Oct 29 22:07:44.076607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.076577 4543 generic.go:356] "Generic (PLEG): container finished" podID="f7557f8b-d351-4d59-87ca-62e277350a0f" containerID="6f3fd21f1be66caa3f3be45f89a1b781fde71dd20501de456ace17a4c5139969" exitCode=0 Oct 29 22:07:44.076712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.076628 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" event={"ID":"f7557f8b-d351-4d59-87ca-62e277350a0f","Type":"ContainerDied","Data":"6f3fd21f1be66caa3f3be45f89a1b781fde71dd20501de456ace17a4c5139969"} Oct 29 22:07:44.118373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.118292 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" podStartSLOduration=2.118278481 podStartE2EDuration="2.118278481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:44.115578875 +0000 UTC m=+1525.857509532" watchObservedRunningTime="2026-10-29 22:07:44.118278481 +0000 UTC m=+1525.860209153" Oct 29 22:07:44.148112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.147427 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:44.148112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.147482 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:44.174029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:44.173947 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" podStartSLOduration=2.173930625 podStartE2EDuration="2.173930625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:44.173241696 +0000 UTC m=+1525.915172369" watchObservedRunningTime="2026-10-29 22:07:44.173930625 +0000 UTC m=+1525.915861297" Oct 29 22:07:45.087207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.087103 4543 generic.go:356] "Generic (PLEG): container finished" podID="e900d3c2-52a2-44af-bc16-534c94a35041" containerID="5f9be9b7cd3a5c8232b095c4f125a2944e23d58dde7cc4fae17f209830d0f9e4" exitCode=0 Oct 29 22:07:45.087207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.087194 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" event={"ID":"e900d3c2-52a2-44af-bc16-534c94a35041","Type":"ContainerDied","Data":"5f9be9b7cd3a5c8232b095c4f125a2944e23d58dde7cc4fae17f209830d0f9e4"} Oct 29 22:07:45.088990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.088963 4543 generic.go:356] "Generic (PLEG): container finished" podID="3e155b22-7abe-42d4-9100-ee19e3083836" containerID="c8f03a4a9736784e7ee8d759f2e463afea6b7f5c006489d779ffeb3760dd088c" exitCode=0 Oct 29 22:07:45.089131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.089026 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" event={"ID":"3e155b22-7abe-42d4-9100-ee19e3083836","Type":"ContainerDied","Data":"c8f03a4a9736784e7ee8d759f2e463afea6b7f5c006489d779ffeb3760dd088c"} Oct 29 22:07:45.139651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.139614 4543 patch_prober.go:29] interesting pod/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 22:07:45.139828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.139686 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 22:07:45.275542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.275515 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:45.328374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.328342 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\") pod \"f7557f8b-d351-4d59-87ca-62e277350a0f\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " Oct 29 22:07:45.328568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.328523 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\") pod \"f7557f8b-d351-4d59-87ca-62e277350a0f\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " Oct 29 22:07:45.328681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.328659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca" pod "f7557f8b-d351-4d59-87ca-62e277350a0f" (UID: "f7557f8b-d351-4d59-87ca-62e277350a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:45.328758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.328679 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\" (UniqueName: \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\") pod \"f7557f8b-d351-4d59-87ca-62e277350a0f\" (UID: \"f7557f8b-d351-4d59-87ca-62e277350a0f\") " Oct 29 22:07:45.329469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.329445 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:45.331064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.330895 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca" pod "f7557f8b-d351-4d59-87ca-62e277350a0f" (UID: "f7557f8b-d351-4d59-87ca-62e277350a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:45.335042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.333660 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz" pod "f7557f8b-d351-4d59-87ca-62e277350a0f" (UID: "f7557f8b-d351-4d59-87ca-62e277350a0f"). InnerVolumeSpecName "kube-api-access-t9hzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:45.430750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.430722 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hzz\" (UniqueName: \"kubernetes.io/projected/f7557f8b-d351-4d59-87ca-62e277350a0f-kube-api-access-t9hzz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:45.430857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.430753 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7557f8b-d351-4d59-87ca-62e277350a0f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:45.908059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.908031 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:07:45.908233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:45.908173 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:07:46.098280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.098253 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" Oct 29 22:07:46.098280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.098269 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k" event={"ID":"f7557f8b-d351-4d59-87ca-62e277350a0f","Type":"ContainerDied","Data":"f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f"} Oct 29 22:07:46.098733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.098319 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f18edbd8aa24eac93d2f9283a78b8920912375de872d103464545b91d7b57d3f" Oct 29 22:07:46.147272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.147241 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:07:46.149565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.149515 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:07:46.342052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.342029 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:46.347054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.347007 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:46.445275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445190 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\") pod \"3e155b22-7abe-42d4-9100-ee19e3083836\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\") pod \"3e155b22-7abe-42d4-9100-ee19e3083836\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\" (UniqueName: \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\") pod \"3e155b22-7abe-42d4-9100-ee19e3083836\" (UID: \"3e155b22-7abe-42d4-9100-ee19e3083836\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445594 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\") pod \"e900d3c2-52a2-44af-bc16-534c94a35041\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445651 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\") pod \"e900d3c2-52a2-44af-bc16-534c94a35041\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445730 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\" (UniqueName: \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\") pod \"e900d3c2-52a2-44af-bc16-534c94a35041\" (UID: \"e900d3c2-52a2-44af-bc16-534c94a35041\") " Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.445836 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca" pod "3e155b22-7abe-42d4-9100-ee19e3083836" (UID: "3e155b22-7abe-42d4-9100-ee19e3083836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:46.446702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.446341 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.448711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.448674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca" pod "3e155b22-7abe-42d4-9100-ee19e3083836" (UID: "3e155b22-7abe-42d4-9100-ee19e3083836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:46.459118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.451089 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca" pod "e900d3c2-52a2-44af-bc16-534c94a35041" (UID: "e900d3c2-52a2-44af-bc16-534c94a35041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:46.459118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.451451 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf" pod "e900d3c2-52a2-44af-bc16-534c94a35041" (UID: "e900d3c2-52a2-44af-bc16-534c94a35041"). InnerVolumeSpecName "kube-api-access-hg7zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:46.459118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.451486 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j" pod "3e155b22-7abe-42d4-9100-ee19e3083836" (UID: "3e155b22-7abe-42d4-9100-ee19e3083836"). InnerVolumeSpecName "kube-api-access-wjs5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:46.459118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.453507 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca" pod "e900d3c2-52a2-44af-bc16-534c94a35041" (UID: "e900d3c2-52a2-44af-bc16-534c94a35041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:46.548189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.548159 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.548189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.548189 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e900d3c2-52a2-44af-bc16-534c94a35041-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.548356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.548202 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7zf\" (UniqueName: \"kubernetes.io/projected/e900d3c2-52a2-44af-bc16-534c94a35041-kube-api-access-hg7zf\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.548356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.548214 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e155b22-7abe-42d4-9100-ee19e3083836-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.548356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:46.548226 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjs5j\" (UniqueName: \"kubernetes.io/projected/3e155b22-7abe-42d4-9100-ee19e3083836-kube-api-access-wjs5j\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:47.107634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.107593 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" event={"ID":"e900d3c2-52a2-44af-bc16-534c94a35041","Type":"ContainerDied","Data":"7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064"} Oct 29 22:07:47.107634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.107620 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r" Oct 29 22:07:47.108238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.107655 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7864c540aec20b32ffce9e5d879b78ceeff23cef2470aece4aa020e7289bb064" Oct 29 22:07:47.109344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.109324 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" Oct 29 22:07:47.109486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.109362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t" event={"ID":"3e155b22-7abe-42d4-9100-ee19e3083836","Type":"ContainerDied","Data":"a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5"} Oct 29 22:07:47.109486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:47.109405 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fe73315dc544ce79b9d319dfc3e53710f572bccd3a4a58bf27171d929c41d5" Oct 29 22:07:54.211542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:54.211509 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:07:54.250757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:54.250722 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:07:57.046350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.046313 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:07:57.046945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.046606 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" podUID="1e38e3fb-56ca-4230-b182-5a8301badff4" containerName="payload-processing" containerID="cri-o://069ae733561f5a8a651adf13dd0f3def42759fd4e6934e28cbf0cb4d4a0092c2" gracePeriod=30 Oct 29 22:07:57.064487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.064311 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:07:57.065165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.064544 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" podUID="e6010040-9244-4929-a78d-124820946161" containerName="payload-pre-processing" containerID="cri-o://7611221da94f64f06b213cdac611c7c23d129aac6c6887d27e8c8ffe5085c7b5" gracePeriod=30 Oct 29 22:07:57.215869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.215831 4543 generic.go:356] "Generic (PLEG): container finished" podID="e6010040-9244-4929-a78d-124820946161" containerID="7611221da94f64f06b213cdac611c7c23d129aac6c6887d27e8c8ffe5085c7b5" exitCode=0 Oct 29 22:07:57.216061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.216024 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" event={"ID":"e6010040-9244-4929-a78d-124820946161","Type":"ContainerDied","Data":"7611221da94f64f06b213cdac611c7c23d129aac6c6887d27e8c8ffe5085c7b5"} Oct 29 22:07:57.218668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.218637 4543 generic.go:356] "Generic (PLEG): container finished" podID="1e38e3fb-56ca-4230-b182-5a8301badff4" containerID="069ae733561f5a8a651adf13dd0f3def42759fd4e6934e28cbf0cb4d4a0092c2" exitCode=0 Oct 29 22:07:57.218840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.218693 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" event={"ID":"1e38e3fb-56ca-4230-b182-5a8301badff4","Type":"ContainerDied","Data":"069ae733561f5a8a651adf13dd0f3def42759fd4e6934e28cbf0cb4d4a0092c2"} Oct 29 22:07:57.485708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.485670 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:07:57.487058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.487038 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:07:57.494529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.494508 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\" (UniqueName: \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\") pod \"e6010040-9244-4929-a78d-124820946161\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " Oct 29 22:07:57.494651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.494547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\") pod \"e6010040-9244-4929-a78d-124820946161\" (UID: \"e6010040-9244-4929-a78d-124820946161\") " Oct 29 22:07:57.494736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.494721 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\" (UniqueName: \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\") pod \"1e38e3fb-56ca-4230-b182-5a8301badff4\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " Oct 29 22:07:57.494975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.494943 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\") pod \"1e38e3fb-56ca-4230-b182-5a8301badff4\" (UID: \"1e38e3fb-56ca-4230-b182-5a8301badff4\") " Oct 29 22:07:57.494975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.494970 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume" pod "e6010040-9244-4929-a78d-124820946161" (UID: "e6010040-9244-4929-a78d-124820946161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:57.495566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.495544 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6010040-9244-4929-a78d-124820946161-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.495667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.495579 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume" pod "1e38e3fb-56ca-4230-b182-5a8301badff4" (UID: "1e38e3fb-56ca-4230-b182-5a8301badff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:57.497014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.496985 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt" pod "1e38e3fb-56ca-4230-b182-5a8301badff4" (UID: "1e38e3fb-56ca-4230-b182-5a8301badff4"). InnerVolumeSpecName "kube-api-access-9pfrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:57.497157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.497027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t" pod "e6010040-9244-4929-a78d-124820946161" (UID: "e6010040-9244-4929-a78d-124820946161"). InnerVolumeSpecName "kube-api-access-rwz4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:57.596879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.596780 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwz4t\" (UniqueName: \"kubernetes.io/projected/e6010040-9244-4929-a78d-124820946161-kube-api-access-rwz4t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.597192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.597171 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfrt\" (UniqueName: \"kubernetes.io/projected/1e38e3fb-56ca-4230-b182-5a8301badff4-kube-api-access-9pfrt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.597356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.597339 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e38e3fb-56ca-4230-b182-5a8301badff4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:57.676874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.676827 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k"] Oct 29 22:07:57.680633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:57.680604 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b16b28-pxz6k"] Oct 29 22:07:57.962491 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:07:57.962454 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:07:58.228518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.228443 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" event={"ID":"e6010040-9244-4929-a78d-124820946161","Type":"ContainerDied","Data":"d2d642cfa6f827b465faf6821e8bf2dd331b091184fb2d6e687692a6d6306bec"} Oct 29 22:07:58.228518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.228508 4543 scope.go:122] "RemoveContainer" containerID="7611221da94f64f06b213cdac611c7c23d129aac6c6887d27e8c8ffe5085c7b5" Oct 29 22:07:58.228518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.228457 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n" Oct 29 22:07:58.230159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.230137 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" event={"ID":"1e38e3fb-56ca-4230-b182-5a8301badff4","Type":"ContainerDied","Data":"eac4cf6f6db60410468bbc19bd0fab04b6fcf940947af4cd07aaab8bf0fa4150"} Oct 29 22:07:58.230268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.230150 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj" Oct 29 22:07:58.240989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.240972 4543 scope.go:122] "RemoveContainer" containerID="069ae733561f5a8a651adf13dd0f3def42759fd4e6934e28cbf0cb4d4a0092c2" Oct 29 22:07:58.287264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.287236 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:07:58.300345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.300326 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b16b28-674cf8b68b-db5dj"] Oct 29 22:07:58.339166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.339141 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:07:58.343919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.343899 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b16b28-57fb56d87-wwg8n"] Oct 29 22:07:58.386580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.386556 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e38e3fb-56ca-4230-b182-5a8301badff4" path="/var/lib/kubelet/pods/1e38e3fb-56ca-4230-b182-5a8301badff4/volumes" Oct 29 22:07:58.386956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.386942 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6010040-9244-4929-a78d-124820946161" path="/var/lib/kubelet/pods/e6010040-9244-4929-a78d-124820946161/volumes" Oct 29 22:07:58.387421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:58.387406 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7557f8b-d351-4d59-87ca-62e277350a0f" path="/var/lib/kubelet/pods/f7557f8b-d351-4d59-87ca-62e277350a0f/volumes" Oct 29 22:07:59.211984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.211948 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" podUID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" containerName="payload-processing" containerID="cri-o://326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be" gracePeriod=25 Oct 29 22:07:59.251691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.251658 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" podUID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" containerName="payload-pre-processing" containerID="cri-o://b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18" gracePeriod=25 Oct 29 22:07:59.572614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.572547 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:07:59.591734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.591710 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:07:59.623776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.623748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\") pod \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " Oct 29 22:07:59.623906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.623868 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\") pod \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " Oct 29 22:07:59.623969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.623908 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\" (UniqueName: \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\") pod \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\" (UID: \"8d2dbc2c-36be-4570-ae5b-354a8fc445ca\") " Oct 29 22:07:59.623969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.623960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\" (UniqueName: \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\") pod \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\" (UID: \"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb\") " Oct 29 22:07:59.624272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.624245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume" pod "8d2dbc2c-36be-4570-ae5b-354a8fc445ca" (UID: "8d2dbc2c-36be-4570-ae5b-354a8fc445ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:59.624346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.624287 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume" pod "e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" (UID: "e4c3d7b2-7ce8-436e-b970-ca60ad9446cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:59.624630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.624605 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:59.624747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.624634 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:59.626897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.626872 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2" pod "8d2dbc2c-36be-4570-ae5b-354a8fc445ca" (UID: "8d2dbc2c-36be-4570-ae5b-354a8fc445ca"). InnerVolumeSpecName "kube-api-access-7xzs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:59.626989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.626909 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn" pod "e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" (UID: "e4c3d7b2-7ce8-436e-b970-ca60ad9446cb"). InnerVolumeSpecName "kube-api-access-9z5dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:59.726195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.726158 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzs2\" (UniqueName: \"kubernetes.io/projected/8d2dbc2c-36be-4570-ae5b-354a8fc445ca-kube-api-access-7xzs2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:07:59.726195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:07:59.726196 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5dn\" (UniqueName: \"kubernetes.io/projected/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb-kube-api-access-9z5dn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:00.248627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.248596 4543 generic.go:356] "Generic (PLEG): container finished" podID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" containerID="326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be" exitCode=0 Oct 29 22:08:00.248785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.248643 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" Oct 29 22:08:00.248785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.248690 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" event={"ID":"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb","Type":"ContainerDied","Data":"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be"} Oct 29 22:08:00.248785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.248780 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86" event={"ID":"e4c3d7b2-7ce8-436e-b970-ca60ad9446cb","Type":"ContainerDied","Data":"7665e6f5ece27e354a0691a401f59a7b2d5367da940efce2749c3618c0fb903a"} Oct 29 22:08:00.248966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.248809 4543 scope.go:122] "RemoveContainer" containerID="326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be" Oct 29 22:08:00.250405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.250367 4543 generic.go:356] "Generic (PLEG): container finished" podID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" containerID="b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18" exitCode=0 Oct 29 22:08:00.250515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.250415 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" Oct 29 22:08:00.250515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.250464 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" event={"ID":"8d2dbc2c-36be-4570-ae5b-354a8fc445ca","Type":"ContainerDied","Data":"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18"} Oct 29 22:08:00.250515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.250507 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j" event={"ID":"8d2dbc2c-36be-4570-ae5b-354a8fc445ca","Type":"ContainerDied","Data":"e0fc1cfe369d07da6e4cd35916b6704f1dbaa03a4217ab1e5bf6502d53d247cf"} Oct 29 22:08:00.261184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.260946 4543 scope.go:122] "RemoveContainer" containerID="326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be" Oct 29 22:08:00.261431 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:00.261221 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be\": container with ID starting with 326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be not found: ID does not exist" containerID="326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be" Oct 29 22:08:00.261431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.261248 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be"} err="failed to get container status \"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be\": rpc error: code = NotFound desc = could not find container \"326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be\": container with ID starting with 326563605ed1d288129b5731d0e17c033340e63d2697e88d50fb59def82783be not found: ID does not exist" Oct 29 22:08:00.261431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.261266 4543 scope.go:122] "RemoveContainer" containerID="b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18" Oct 29 22:08:00.271134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.271116 4543 scope.go:122] "RemoveContainer" containerID="b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18" Oct 29 22:08:00.271402 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:00.271383 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18\": container with ID starting with b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18 not found: ID does not exist" containerID="b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18" Oct 29 22:08:00.271485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.271408 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18"} err="failed to get container status \"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18\": rpc error: code = NotFound desc = could not find container \"b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18\": container with ID starting with b4d28652c7c3c812204213b4bfb624ff1224ccf598a7ac5f6abeb099e9747f18 not found: ID does not exist" Oct 29 22:08:00.309201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.309173 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:08:00.314403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.314378 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bc531-5d48c89cc9-g4l7j"] Oct 29 22:08:00.354208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.354184 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:08:00.357619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.357596 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bc531-6bbb7d675b-5wj86"] Oct 29 22:08:00.387754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.387726 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" path="/var/lib/kubelet/pods/8d2dbc2c-36be-4570-ae5b-354a8fc445ca/volumes" Oct 29 22:08:00.388524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:00.388498 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" path="/var/lib/kubelet/pods/e4c3d7b2-7ce8-436e-b970-ca60ad9446cb/volumes" Oct 29 22:08:18.476696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.476666 4543 generic.go:356] "Generic (PLEG): container finished" podID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerID="a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434" exitCode=0 Oct 29 22:08:18.477267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.476725 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerDied","Data":"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434"} Oct 29 22:08:18.477267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.477163 4543 scope.go:122] "RemoveContainer" containerID="a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434" Oct 29 22:08:18.478439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.478417 4543 generic.go:356] "Generic (PLEG): container finished" podID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerID="de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7" exitCode=0 Oct 29 22:08:18.478570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.478493 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerDied","Data":"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7"} Oct 29 22:08:18.478890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.478867 4543 scope.go:122] "RemoveContainer" containerID="de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7" Oct 29 22:08:18.478989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.478939 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.479030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.479006 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.479066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.479047 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.480601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.480572 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.480715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.480656 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.480715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.480703 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:18.812738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.812667 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n"] Oct 29 22:08:18.813403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813387 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e6010040-9244-4929-a78d-124820946161" containerName="payload-pre-processing" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813405 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6010040-9244-4929-a78d-124820946161" containerName="payload-pre-processing" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813417 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3e155b22-7abe-42d4-9100-ee19e3083836" containerName="revoke-keys" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813422 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e155b22-7abe-42d4-9100-ee19e3083836" containerName="revoke-keys" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813451 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f7557f8b-d351-4d59-87ca-62e277350a0f" containerName="revoke-keys" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813455 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7557f8b-d351-4d59-87ca-62e277350a0f" containerName="revoke-keys" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813464 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1e38e3fb-56ca-4230-b182-5a8301badff4" containerName="payload-processing" Oct 29 22:08:18.813470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813469 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e38e3fb-56ca-4230-b182-5a8301badff4" containerName="payload-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813478 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" containerName="payload-pre-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813484 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" containerName="payload-pre-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813494 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e900d3c2-52a2-44af-bc16-534c94a35041" containerName="revoke-keys" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813498 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e900d3c2-52a2-44af-bc16-534c94a35041" containerName="revoke-keys" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813505 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" containerName="payload-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813509 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" containerName="payload-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813599 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6010040-9244-4929-a78d-124820946161" containerName="payload-pre-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813610 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e155b22-7abe-42d4-9100-ee19e3083836" containerName="revoke-keys" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813620 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7557f8b-d351-4d59-87ca-62e277350a0f" containerName="revoke-keys" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813625 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e38e3fb-56ca-4230-b182-5a8301badff4" containerName="payload-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813631 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2dbc2c-36be-4570-ae5b-354a8fc445ca" containerName="payload-pre-processing" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813637 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e900d3c2-52a2-44af-bc16-534c94a35041" containerName="revoke-keys" Oct 29 22:08:18.813694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.813641 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c3d7b2-7ce8-436e-b970-ca60ad9446cb" containerName="payload-processing" Oct 29 22:08:18.816189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.816173 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.837121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.835177 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:08:18.837121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.835224 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:08:18.837121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.835272 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:08:18.837121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.835374 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:08:18.837862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.837840 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n"] Oct 29 22:08:18.874471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.874443 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.874632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.874613 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.874842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.874819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km674\" (UniqueName: \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.976193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.976154 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km674\" (UniqueName: \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.976377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.976255 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.976377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.976322 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.977179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.977147 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:18.978781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:18.978760 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:19.002123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.002093 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km674\" (UniqueName: \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\") pod \"maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:19.137121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.137056 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:19.409959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.408210 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:08:19.475276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.475201 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n"] Oct 29 22:08:19.494690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.494649 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" event={"ID":"e612621e-1a4e-49ee-8b8e-74b10a305a27","Type":"ContainerStarted","Data":"abcbe3e854e3c0e60ac601b9a06da0dc379d1b7644006581e8f976121f12febc"} Oct 29 22:08:19.497857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.497826 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerStarted","Data":"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44"} Oct 29 22:08:19.501030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.501004 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerStarted","Data":"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c"} Oct 29 22:08:19.551509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.549438 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:08:19.551509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.551154 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:19.551509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.551228 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:19.986002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:19.985966 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:08:20.023736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.023705 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:08:20.378102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.375564 4543 scope.go:122] "RemoveContainer" containerID="eac2cce1d6fc654be4872c0df5c57340fc8170f2844b5504a8a0607ae34b90d8" Oct 29 22:08:20.417240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.417211 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:08:20.417416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.417268 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:08:20.417564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.417535 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:08:20.472102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.439807 4543 scope.go:122] "RemoveContainer" containerID="1a1ae2cbe20b4f8f37a9e6676728b170c55de66195b4e13ad7896dfcd5420575" Oct 29 22:08:20.515850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.515814 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" event={"ID":"e612621e-1a4e-49ee-8b8e-74b10a305a27","Type":"ContainerStarted","Data":"e6b79be0caae321bede3b44234be34f0b9a45fcea5b6b01d7e3d620ac8b32b9a"} Oct 29 22:08:20.520240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.520223 4543 scope.go:122] "RemoveContainer" containerID="85e8ecfdb1fb945e52952a8adedd564fb4366886adcd5efda782fdb32d878eb7" Oct 29 22:08:20.579103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.548041 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:20.579103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.548113 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:20.583042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.583011 4543 scope.go:122] "RemoveContainer" containerID="4aff5a0aa1796655bdc5e8d11cc5b6687058d26f37fa14a0abaf42a3b8793ea1" Oct 29 22:08:20.584542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.584502 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" podStartSLOduration=2.584488018 podStartE2EDuration="2.584488018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:20.57372323 +0000 UTC m=+1562.315653904" watchObservedRunningTime="2026-10-29 22:08:20.584488018 +0000 UTC m=+1562.326418691" Oct 29 22:08:20.593632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:20.593616 4543 scope.go:122] "RemoveContainer" containerID="3d74adf9be66d9890a79fdecc51639514706b13a6777b51edfa9600f120c4d0b" Oct 29 22:08:21.416402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.416368 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:08:21.416683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.416651 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:08:21.420601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.420571 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:08:21.528023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.527983 4543 generic.go:356] "Generic (PLEG): container finished" podID="e612621e-1a4e-49ee-8b8e-74b10a305a27" containerID="e6b79be0caae321bede3b44234be34f0b9a45fcea5b6b01d7e3d620ac8b32b9a" exitCode=0 Oct 29 22:08:21.528594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.528064 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" event={"ID":"e612621e-1a4e-49ee-8b8e-74b10a305a27","Type":"ContainerDied","Data":"e6b79be0caae321bede3b44234be34f0b9a45fcea5b6b01d7e3d620ac8b32b9a"} Oct 29 22:08:21.562475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.562067 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:21.562475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:21.562175 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:22.417214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.417183 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:08:22.417390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.417236 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:08:22.550105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.549450 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:22.550105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.549701 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:22.887405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.887372 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:22.927623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.927544 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\") pod \"e612621e-1a4e-49ee-8b8e-74b10a305a27\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " Oct 29 22:08:22.927761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.927658 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\" (UniqueName: \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\") pod \"e612621e-1a4e-49ee-8b8e-74b10a305a27\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " Oct 29 22:08:22.927761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.927736 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\") pod \"e612621e-1a4e-49ee-8b8e-74b10a305a27\" (UID: \"e612621e-1a4e-49ee-8b8e-74b10a305a27\") " Oct 29 22:08:22.927996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.927960 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca" pod "e612621e-1a4e-49ee-8b8e-74b10a305a27" (UID: "e612621e-1a4e-49ee-8b8e-74b10a305a27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:22.932101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.928370 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:22.949099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:22.946274 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674" pod "e612621e-1a4e-49ee-8b8e-74b10a305a27" (UID: "e612621e-1a4e-49ee-8b8e-74b10a305a27"). InnerVolumeSpecName "kube-api-access-km674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:23.005324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.005282 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca" pod "e612621e-1a4e-49ee-8b8e-74b10a305a27" (UID: "e612621e-1a4e-49ee-8b8e-74b10a305a27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:23.078221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.033386 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km674\" (UniqueName: \"kubernetes.io/projected/e612621e-1a4e-49ee-8b8e-74b10a305a27-kube-api-access-km674\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:23.078221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.078165 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e612621e-1a4e-49ee-8b8e-74b10a305a27-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:23.416574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.416541 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:08:23.416759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.416593 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:08:23.550100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.549441 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:23.550100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.549499 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:23.590718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.590689 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" event={"ID":"e612621e-1a4e-49ee-8b8e-74b10a305a27","Type":"ContainerDied","Data":"abcbe3e854e3c0e60ac601b9a06da0dc379d1b7644006581e8f976121f12febc"} Oct 29 22:08:23.590917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.590905 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abcbe3e854e3c0e60ac601b9a06da0dc379d1b7644006581e8f976121f12febc" Oct 29 22:08:23.591050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:23.591042 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n" Oct 29 22:08:24.416361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:24.416328 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:08:24.416542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:24.416375 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 22:08:24.550101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:24.548457 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:08:24.550101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:24.548520 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 22:08:24.986853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:24.986786 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" podUID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" containerName="payload-processing" containerID="cri-o://f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb" gracePeriod=25 Oct 29 22:08:25.024364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.024323 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" podUID="23d7773e-afe5-4e37-a01b-41043a5a4886" containerName="payload-pre-processing" containerID="cri-o://f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a" gracePeriod=25 Oct 29 22:08:25.335784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.335760 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:08:25.363777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.363753 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:08:25.421835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.421808 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:08:25.421981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.421928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\") pod \"23d7773e-afe5-4e37-a01b-41043a5a4886\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " Oct 29 22:08:25.421981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.421959 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\") pod \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " Oct 29 22:08:25.422066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.421995 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\" (UniqueName: \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\") pod \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\" (UID: \"3d774a86-2104-4b4a-a0ee-366322d1bfb4\") " Oct 29 22:08:25.422066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.422062 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\" (UniqueName: \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\") pod \"23d7773e-afe5-4e37-a01b-41043a5a4886\" (UID: \"23d7773e-afe5-4e37-a01b-41043a5a4886\") " Oct 29 22:08:25.422417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.422387 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume" pod "23d7773e-afe5-4e37-a01b-41043a5a4886" (UID: "23d7773e-afe5-4e37-a01b-41043a5a4886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:25.422417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.422393 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume" pod "3d774a86-2104-4b4a-a0ee-366322d1bfb4" (UID: "3d774a86-2104-4b4a-a0ee-366322d1bfb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:25.423244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.423206 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:08:25.423839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.423817 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d7773e-afe5-4e37-a01b-41043a5a4886-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:25.423941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.423840 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d774a86-2104-4b4a-a0ee-366322d1bfb4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:25.424838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.424802 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm" pod "3d774a86-2104-4b4a-a0ee-366322d1bfb4" (UID: "3d774a86-2104-4b4a-a0ee-366322d1bfb4"). InnerVolumeSpecName "kube-api-access-xfmxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:25.425148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.425124 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn" pod "23d7773e-afe5-4e37-a01b-41043a5a4886" (UID: "23d7773e-afe5-4e37-a01b-41043a5a4886"). InnerVolumeSpecName "kube-api-access-4smgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:25.525640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.525556 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmxm\" (UniqueName: \"kubernetes.io/projected/3d774a86-2104-4b4a-a0ee-366322d1bfb4-kube-api-access-xfmxm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:25.525640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.525591 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smgn\" (UniqueName: \"kubernetes.io/projected/23d7773e-afe5-4e37-a01b-41043a5a4886-kube-api-access-4smgn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:25.553097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.553053 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:08:25.553478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.553454 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:08:25.554250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.554227 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:08:25.607506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.607475 4543 generic.go:356] "Generic (PLEG): container finished" podID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" containerID="f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb" exitCode=0 Oct 29 22:08:25.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.607532 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" Oct 29 22:08:25.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.607569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" event={"ID":"3d774a86-2104-4b4a-a0ee-366322d1bfb4","Type":"ContainerDied","Data":"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb"} Oct 29 22:08:25.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.607619 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n" event={"ID":"3d774a86-2104-4b4a-a0ee-366322d1bfb4","Type":"ContainerDied","Data":"11bd3d5ef22f96c672c9ce4a5c035864b2f3480eaa08ae47f55819e4c7ba397e"} Oct 29 22:08:25.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.607649 4543 scope.go:122] "RemoveContainer" containerID="f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb" Oct 29 22:08:25.609212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.609188 4543 generic.go:356] "Generic (PLEG): container finished" podID="23d7773e-afe5-4e37-a01b-41043a5a4886" containerID="f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a" exitCode=0 Oct 29 22:08:25.609349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.609265 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" Oct 29 22:08:25.609349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.609272 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" event={"ID":"23d7773e-afe5-4e37-a01b-41043a5a4886","Type":"ContainerDied","Data":"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a"} Oct 29 22:08:25.609349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.609322 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp" event={"ID":"23d7773e-afe5-4e37-a01b-41043a5a4886","Type":"ContainerDied","Data":"a74130e3efaaa7b401834806c01503d9e424212dcd6333305ca5a39717ac9827"} Oct 29 22:08:25.620429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.620410 4543 scope.go:122] "RemoveContainer" containerID="f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb" Oct 29 22:08:25.620758 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:25.620735 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb\": container with ID starting with f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb not found: ID does not exist" containerID="f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb" Oct 29 22:08:25.620874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.620766 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb"} err="failed to get container status \"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb\": rpc error: code = NotFound desc = could not find container \"f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb\": container with ID starting with f812ad86ef4355214a1284305b4c2ffb1a507850eee9055979aa50092a7dfdcb not found: ID does not exist" Oct 29 22:08:25.620874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.620783 4543 scope.go:122] "RemoveContainer" containerID="f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a" Oct 29 22:08:25.630922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.630906 4543 scope.go:122] "RemoveContainer" containerID="f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a" Oct 29 22:08:25.631229 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:25.631208 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a\": container with ID starting with f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a not found: ID does not exist" containerID="f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a" Oct 29 22:08:25.631298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.631239 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a"} err="failed to get container status \"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a\": rpc error: code = NotFound desc = could not find container \"f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a\": container with ID starting with f2477542585a6577f18c8f9c0c6ff1c89f89acc7429d5eeff9b73003bf47c99a not found: ID does not exist" Oct 29 22:08:25.737438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.737408 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:08:25.742649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.742620 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c53cc8-6959f4cbd8-rzw4n"] Oct 29 22:08:25.779782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.779720 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:08:25.784831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:25.784798 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c53cc8-74ff6b687f-9rldp"] Oct 29 22:08:26.391300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.391270 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d7773e-afe5-4e37-a01b-41043a5a4886" path="/var/lib/kubelet/pods/23d7773e-afe5-4e37-a01b-41043a5a4886/volumes" Oct 29 22:08:26.391788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.391766 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" path="/var/lib/kubelet/pods/3d774a86-2104-4b4a-a0ee-366322d1bfb4/volumes" Oct 29 22:08:26.609110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609033 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:08:26.609825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609806 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e612621e-1a4e-49ee-8b8e-74b10a305a27" containerName="revoke-keys" Oct 29 22:08:26.609825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609824 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e612621e-1a4e-49ee-8b8e-74b10a305a27" containerName="revoke-keys" Oct 29 22:08:26.610027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609847 4543 cpu_manager.go:414] "containerMap: removing container" podUID="23d7773e-afe5-4e37-a01b-41043a5a4886" containerName="payload-pre-processing" Oct 29 22:08:26.610027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609856 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d7773e-afe5-4e37-a01b-41043a5a4886" containerName="payload-pre-processing" Oct 29 22:08:26.610027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609878 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" containerName="payload-processing" Oct 29 22:08:26.610027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.609883 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" containerName="payload-processing" Oct 29 22:08:26.610027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.610019 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d774a86-2104-4b4a-a0ee-366322d1bfb4" containerName="payload-processing" Oct 29 22:08:26.610318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.610037 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e612621e-1a4e-49ee-8b8e-74b10a305a27" containerName="revoke-keys" Oct 29 22:08:26.610318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.610045 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d7773e-afe5-4e37-a01b-41043a5a4886" containerName="payload-pre-processing" Oct 29 22:08:26.613844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.613828 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.627695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.627676 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:08:26.631017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.630999 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.640700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.640660 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:08:26.644583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.644516 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.644797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.644769 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.644915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.644882 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646211 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646305 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgcj6\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646411 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646527 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.646762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.646603 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.658895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.658867 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:08:26.747731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747700 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.747871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747744 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.747871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747768 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747890 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747922 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.747942 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748035 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748059 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748131 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748159 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748173 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748217 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748258 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgzr\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748314 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcj6\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748396 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748454 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748478 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.748777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748508 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748614 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.748777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748706 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.749024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.748997 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.749160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.749121 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.750951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.750934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.751129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.751113 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.773836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.773812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.773931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.773910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgcj6\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.850474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850451 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850486 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850564 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850595 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850667 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850689 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.850897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.850849 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgzr\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.851137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.851112 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.851299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.851203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.851432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.851404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.851575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.851416 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.851653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.851629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.852969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.852952 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.853752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.853736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.876497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.876470 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.876579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.876501 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgzr\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:26.929612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.929566 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:26.952741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:26.952715 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:27.258229 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:08:27.258195 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1af3ef_ce1a_4439_8310_b43f160815eb.slice/crio-4cf9dd597db0c30fd631d1707a03afaf4bc08e37d38f4f79e0862e29feebd434 WatchSource:0}: Error finding container 4cf9dd597db0c30fd631d1707a03afaf4bc08e37d38f4f79e0862e29feebd434: Status 404 returned error can't find the container with id 4cf9dd597db0c30fd631d1707a03afaf4bc08e37d38f4f79e0862e29feebd434 Oct 29 22:08:27.258785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.258756 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:08:27.260775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.260741 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.260925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.260803 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.260925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.260844 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.301969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.301942 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:08:27.302843 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:08:27.302816 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff51eaed_e440_4001_a6dc_0f067a99d86c.slice/crio-7dfde03c083343dff2a1378715c440159c0aade9d80572cb55f3685fbee0cad8 WatchSource:0}: Error finding container 7dfde03c083343dff2a1378715c440159c0aade9d80572cb55f3685fbee0cad8: Status 404 returned error can't find the container with id 7dfde03c083343dff2a1378715c440159c0aade9d80572cb55f3685fbee0cad8 Oct 29 22:08:27.305578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.305545 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.305675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.305659 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.305723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.305693 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:27.633975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.633887 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" event={"ID":"0b1af3ef-ce1a-4439-8310-b43f160815eb","Type":"ContainerStarted","Data":"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc"} Oct 29 22:08:27.634560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.634534 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" event={"ID":"0b1af3ef-ce1a-4439-8310-b43f160815eb","Type":"ContainerStarted","Data":"4cf9dd597db0c30fd631d1707a03afaf4bc08e37d38f4f79e0862e29feebd434"} Oct 29 22:08:27.636291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.636267 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" event={"ID":"ff51eaed-e440-4001-a6dc-0f067a99d86c","Type":"ContainerStarted","Data":"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778"} Oct 29 22:08:27.636405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.636310 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" event={"ID":"ff51eaed-e440-4001-a6dc-0f067a99d86c","Type":"ContainerStarted","Data":"7dfde03c083343dff2a1378715c440159c0aade9d80572cb55f3685fbee0cad8"} Oct 29 22:08:27.683961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.683889 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podStartSLOduration=1.683856049 podStartE2EDuration="1.683856049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:27.682182316 +0000 UTC m=+1569.424113014" watchObservedRunningTime="2026-10-29 22:08:27.683856049 +0000 UTC m=+1569.425786725" Oct 29 22:08:27.733886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.733825 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podStartSLOduration=1.73380598 podStartE2EDuration="1.73380598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:27.733406098 +0000 UTC m=+1569.475336771" watchObservedRunningTime="2026-10-29 22:08:27.73380598 +0000 UTC m=+1569.475736654" Oct 29 22:08:27.930294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.930221 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:27.952995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:27.952972 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:27.962460 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:27.962430 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:08:28.948237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:28.948194 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:28.948901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:28.948866 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:28.955884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:28.955857 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:28.956140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:28.956112 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:29.937473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:29.937383 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:29.937754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:29.937727 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:29.955396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:29.955368 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:29.955881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:29.955856 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:30.931436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:30.931400 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:30.931715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:30.931687 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:30.953434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:30.953403 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:30.953663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:30.953638 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:31.945101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:31.930405 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:31.945101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:31.930461 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:31.953473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:31.953441 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:31.953714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:31.953685 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:32.933104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:32.932531 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:32.933104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:32.932607 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:32.969102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:32.967576 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:32.969102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:32.967636 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:33.934110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:33.933484 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:33.934110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:33.933554 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:33.956099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:33.955948 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:33.956099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:33.956001 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:34.947964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:34.947917 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 29 22:08:34.948590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:34.948542 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 29 22:08:34.970503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:34.970465 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 29 22:08:34.970683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:34.970523 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 29 22:08:35.121778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.119170 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:35.123734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.122042 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:08:35.195200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.195157 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:08:35.310125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.309168 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:35.445730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.445685 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:35.505102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.502641 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:35.612895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.588533 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 29 22:08:35.770102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:35.741419 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:08:36.673506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.673469 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:36.675350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.675324 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:36.681388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.681364 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:08:36.811457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.810557 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:36.812323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.812256 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:36.814511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:36.814476 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:08:38.663158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:38.663126 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n"] Oct 29 22:08:38.669886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:38.669861 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9bb81b-zzw6n"] Oct 29 22:08:40.390788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:40.390755 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e612621e-1a4e-49ee-8b8e-74b10a305a27" path="/var/lib/kubelet/pods/e612621e-1a4e-49ee-8b8e-74b10a305a27/volumes" Oct 29 22:08:43.497728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:43.497695 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r"] Oct 29 22:08:43.514475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:43.514443 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5bc531-z5q6r"] Oct 29 22:08:44.372562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.372461 4543 generic.go:356] "Generic (PLEG): container finished" podID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerID="9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b" exitCode=0 Oct 29 22:08:44.372780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.372677 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerDied","Data":"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b"} Oct 29 22:08:44.373260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.373233 4543 scope.go:122] "RemoveContainer" containerID="9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b" Oct 29 22:08:44.375357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.375327 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:44.375435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.375399 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:44.375435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.375430 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:08:44.386275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:44.386252 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e900d3c2-52a2-44af-bc16-534c94a35041" path="/var/lib/kubelet/pods/e900d3c2-52a2-44af-bc16-534c94a35041/volumes" Oct 29 22:08:45.383383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:45.383348 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerStarted","Data":"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3"} Oct 29 22:08:45.438099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:45.431040 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:08:45.438099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:45.431563 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:45.438099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:45.431713 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:46.448100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:46.433447 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:46.448100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:46.433505 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:46.826695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:46.826656 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:08:46.826914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:46.826889 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" containerID="cri-o://51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1" gracePeriod=30 Oct 29 22:08:47.456019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.455501 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:47.456019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.455554 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:47.460393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.460366 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2"] Oct 29 22:08:47.464992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.464955 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.473257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.473233 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2"] Oct 29 22:08:47.482368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.482348 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:08:47.482485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.482427 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:08:47.504391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.504368 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:08:47.504475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.504411 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:08:47.521842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.521819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhrq\" (UniqueName: \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.521994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.521976 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.522150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.522130 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.623391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.623359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhrq\" (UniqueName: \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.623553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.623498 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.623615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.623600 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.624373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.624303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.626147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.626124 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.658895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.658837 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhrq\" (UniqueName: \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:47.784128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:47.784097 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:48.122348 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:08:48.122300 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabf5345_6434_484f_b4e8_100ebf55a2cb.slice/crio-b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531 WatchSource:0}: Error finding container b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531: Status 404 returned error can't find the container with id b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531 Oct 29 22:08:48.123294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.123264 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2"] Oct 29 22:08:48.406427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.406398 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" event={"ID":"babf5345-6434-484f-b4e8-100ebf55a2cb","Type":"ContainerStarted","Data":"b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531"} Oct 29 22:08:48.449862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.449830 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:48.450556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.450528 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:48.854103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.852615 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:08:48.854103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:48.852672 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:08:49.415769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:49.415738 4543 generic.go:356] "Generic (PLEG): container finished" podID="babf5345-6434-484f-b4e8-100ebf55a2cb" containerID="af0a910a09d0492fa7923a717431f8b415802f736f59ab4d8b19419b97afee52" exitCode=0 Oct 29 22:08:49.415907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:49.415817 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" event={"ID":"babf5345-6434-484f-b4e8-100ebf55a2cb","Type":"ContainerDied","Data":"af0a910a09d0492fa7923a717431f8b415802f736f59ab4d8b19419b97afee52"} Oct 29 22:08:49.443590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:49.443562 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:49.447599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:49.447569 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:50.453228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.453193 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:08:50.453699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.453263 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 29 22:08:50.632835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.632798 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:50.700880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.700795 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\") pod \"babf5345-6434-484f-b4e8-100ebf55a2cb\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " Oct 29 22:08:50.701167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.701149 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\") pod \"babf5345-6434-484f-b4e8-100ebf55a2cb\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " Oct 29 22:08:50.701285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.701205 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\" (UniqueName: \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\") pod \"babf5345-6434-484f-b4e8-100ebf55a2cb\" (UID: \"babf5345-6434-484f-b4e8-100ebf55a2cb\") " Oct 29 22:08:50.701955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.701672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca" pod "babf5345-6434-484f-b4e8-100ebf55a2cb" (UID: "babf5345-6434-484f-b4e8-100ebf55a2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:50.701955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.701919 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:50.709118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.707494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca" pod "babf5345-6434-484f-b4e8-100ebf55a2cb" (UID: "babf5345-6434-484f-b4e8-100ebf55a2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:50.709118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.707635 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq" pod "babf5345-6434-484f-b4e8-100ebf55a2cb" (UID: "babf5345-6434-484f-b4e8-100ebf55a2cb"). InnerVolumeSpecName "kube-api-access-xjhrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:50.803846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.803814 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhrq\" (UniqueName: \"kubernetes.io/projected/babf5345-6434-484f-b4e8-100ebf55a2cb-kube-api-access-xjhrq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:50.803846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:50.803849 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babf5345-6434-484f-b4e8-100ebf55a2cb-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:51.461352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.461316 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:08:51.463750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.463719 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:08:51.510419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.510387 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:08:51.556104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.556040 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" Oct 29 22:08:51.559488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.559455 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2" event={"ID":"babf5345-6434-484f-b4e8-100ebf55a2cb","Type":"ContainerDied","Data":"b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531"} Oct 29 22:08:51.559845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:51.559811 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1f6c8a0026befcfd51b62bb7cd33f16f083ad8fd0702c4ead63bdc996f99531" Oct 29 22:08:52.042914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.042894 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.173577 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.173809 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.173843 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.173898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.173975 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.174014 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.174148 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.174184 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.176149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.174227 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\") pod \"9dea599f-9164-4a38-b80d-0de739784091\" (UID: \"9dea599f-9164-4a38-b80d-0de739784091\") " Oct 29 22:08:52.182306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.182269 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:52.184539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.184046 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:52.185554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.185498 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:52.185952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.185931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:52.234727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.226170 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:52.296171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.289403 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.296888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.296862 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.297096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.297062 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.297210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.297197 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dea599f-9164-4a38-b80d-0de739784091-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.297298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.297288 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.317338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.317245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "kube-api-access-z8vsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:52.326900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.326740 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:52.326900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.326837 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:08:52.346295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.346203 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy" pod "9dea599f-9164-4a38-b80d-0de739784091" (UID: "9dea599f-9164-4a38-b80d-0de739784091"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:52.398880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.398850 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dea599f-9164-4a38-b80d-0de739784091-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.398880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.398878 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.399130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.398888 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8vsq\" (UniqueName: \"kubernetes.io/projected/9dea599f-9164-4a38-b80d-0de739784091-kube-api-access-z8vsq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.399130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.398896 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dea599f-9164-4a38-b80d-0de739784091-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:52.535105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.535052 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:08:52.535952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.535897 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" containerID="cri-o://2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494" gracePeriod=30 Oct 29 22:08:52.569901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.569863 4543 generic.go:356] "Generic (PLEG): container finished" podID="9dea599f-9164-4a38-b80d-0de739784091" containerID="51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1" exitCode=0 Oct 29 22:08:52.570041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.569928 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" Oct 29 22:08:52.570041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.569943 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerDied","Data":"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1"} Oct 29 22:08:52.570041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.570002 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh" event={"ID":"9dea599f-9164-4a38-b80d-0de739784091","Type":"ContainerDied","Data":"c1513f90b1e6fee53e2495681748bb58599e5175e175d21564c9098845627dc5"} Oct 29 22:08:52.570041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.570029 4543 scope.go:122] "RemoveContainer" containerID="51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1" Oct 29 22:08:52.585994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.585971 4543 scope.go:122] "RemoveContainer" containerID="ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0" Oct 29 22:08:52.595774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.595757 4543 scope.go:122] "RemoveContainer" containerID="51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1" Oct 29 22:08:52.596038 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:52.596017 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1\": container with ID starting with 51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1 not found: ID does not exist" containerID="51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1" Oct 29 22:08:52.596159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.596047 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1"} err="failed to get container status \"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1\": rpc error: code = NotFound desc = could not find container \"51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1\": container with ID starting with 51197188b45328db587e9efbfb47e0fcfd8203fbb17561195b6c7c98759b1cf1 not found: ID does not exist" Oct 29 22:08:52.596159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.596069 4543 scope.go:122] "RemoveContainer" containerID="ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0" Oct 29 22:08:52.596349 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:52.596329 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0\": container with ID starting with ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0 not found: ID does not exist" containerID="ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0" Oct 29 22:08:52.596390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.596356 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0"} err="failed to get container status \"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0\": rpc error: code = NotFound desc = could not find container \"ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0\": container with ID starting with ec012ae37077dff4165ad16f6509aeff07d2f1587f388b6c9a977b034a89a1b0 not found: ID does not exist" Oct 29 22:08:52.629064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.629029 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:08:52.645363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:52.645335 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9bb81b-openshift-default-7dc998d95c-pmxhh"] Oct 29 22:08:53.122756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.122728 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn"] Oct 29 22:08:53.123466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123427 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123445 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123457 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123463 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123503 4543 cpu_manager.go:414] "containerMap: removing container" podUID="babf5345-6434-484f-b4e8-100ebf55a2cb" containerName="revoke-keys" Oct 29 22:08:53.123751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123509 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babf5345-6434-484f-b4e8-100ebf55a2cb" containerName="revoke-keys" Oct 29 22:08:53.123751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123638 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123647 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dea599f-9164-4a38-b80d-0de739784091" containerName="istio-proxy" Oct 29 22:08:53.123751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.123665 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="babf5345-6434-484f-b4e8-100ebf55a2cb" containerName="revoke-keys" Oct 29 22:08:53.127246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.127224 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.144871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.144664 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:08:53.144871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.144824 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:08:53.146539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.146517 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn"] Oct 29 22:08:53.211813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.211785 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.212005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.211984 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.212169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.212150 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcncq\" (UniqueName: \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.234018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.233968 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:08:53.234141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.234129 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:08:53.313822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.313796 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.313922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.313877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcncq\" (UniqueName: \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.314018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.314002 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.314508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.314483 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.316502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.316480 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.339326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.339302 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcncq\" (UniqueName: \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\") pod \"maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.440747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.440699 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:53.757664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:53.757640 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn"] Oct 29 22:08:53.759299 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:08:53.759272 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c79b9f_2563_46d1_ad53_c9a07337826a.slice/crio-c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a WatchSource:0}: Error finding container c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a: Status 404 returned error can't find the container with id c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a Oct 29 22:08:54.387338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:54.387265 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dea599f-9164-4a38-b80d-0de739784091" path="/var/lib/kubelet/pods/9dea599f-9164-4a38-b80d-0de739784091/volumes" Oct 29 22:08:54.590855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:54.590816 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" event={"ID":"d2c79b9f-2563-46d1-ad53-c9a07337826a","Type":"ContainerStarted","Data":"783e751f935d842c76d4f6b73b7dc3fdd23154bd8f33809a8f7ff10f43a3ca71"} Oct 29 22:08:54.591002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:54.590906 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" event={"ID":"d2c79b9f-2563-46d1-ad53-c9a07337826a","Type":"ContainerStarted","Data":"c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a"} Oct 29 22:08:54.635993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:54.635940 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" podStartSLOduration=1.635924949 podStartE2EDuration="1.635924949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:54.634108008 +0000 UTC m=+1596.376038676" watchObservedRunningTime="2026-10-29 22:08:54.635924949 +0000 UTC m=+1596.377855624" Oct 29 22:08:55.601012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:55.600982 4543 generic.go:356] "Generic (PLEG): container finished" podID="d2c79b9f-2563-46d1-ad53-c9a07337826a" containerID="783e751f935d842c76d4f6b73b7dc3fdd23154bd8f33809a8f7ff10f43a3ca71" exitCode=0 Oct 29 22:08:55.601405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:55.601064 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" event={"ID":"d2c79b9f-2563-46d1-ad53-c9a07337826a","Type":"ContainerDied","Data":"783e751f935d842c76d4f6b73b7dc3fdd23154bd8f33809a8f7ff10f43a3ca71"} Oct 29 22:08:56.769035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.769011 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:56.863649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.863565 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\") pod \"d2c79b9f-2563-46d1-ad53-c9a07337826a\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " Oct 29 22:08:56.863801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.863663 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\" (UniqueName: \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\") pod \"d2c79b9f-2563-46d1-ad53-c9a07337826a\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " Oct 29 22:08:56.863801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.863754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\") pod \"d2c79b9f-2563-46d1-ad53-c9a07337826a\" (UID: \"d2c79b9f-2563-46d1-ad53-c9a07337826a\") " Oct 29 22:08:56.864546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.864452 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca" pod "d2c79b9f-2563-46d1-ad53-c9a07337826a" (UID: "d2c79b9f-2563-46d1-ad53-c9a07337826a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:56.866969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.866939 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca" pod "d2c79b9f-2563-46d1-ad53-c9a07337826a" (UID: "d2c79b9f-2563-46d1-ad53-c9a07337826a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:08:56.867465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.867442 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq" pod "d2c79b9f-2563-46d1-ad53-c9a07337826a" (UID: "d2c79b9f-2563-46d1-ad53-c9a07337826a"). InnerVolumeSpecName "kube-api-access-mcncq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:56.965903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.965874 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:56.965903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.965902 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcncq\" (UniqueName: \"kubernetes.io/projected/d2c79b9f-2563-46d1-ad53-c9a07337826a-kube-api-access-mcncq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:56.966089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:56.965917 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c79b9f-2563-46d1-ad53-c9a07337826a-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.618113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.618088 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" Oct 29 22:08:57.618245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.618088 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn" event={"ID":"d2c79b9f-2563-46d1-ad53-c9a07337826a","Type":"ContainerDied","Data":"c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a"} Oct 29 22:08:57.618245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.618211 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85541f4fc50f1314c9d39b0911b96d2d771a848a04b508663030cf0c615b79a" Oct 29 22:08:57.868258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.868234 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:08:57.964407 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:57.964371 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:08:57.984738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.984711 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.984937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.984764 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.984937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.984879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.984976 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985146 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985150 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:57.985227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985197 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985246 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:08:57.985360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985335 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985363 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:57.985395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985373 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:57.985456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985403 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.985456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985419 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:57.985456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.985438 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\") pod \"3251f60e-47f1-4fa7-abe3-370ed29d9549\" (UID: \"3251f60e-47f1-4fa7-abe3-370ed29d9549\") " Oct 29 22:08:57.986348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.986313 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.986348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.986343 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.986532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.986356 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.986532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.986369 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3251f60e-47f1-4fa7-abe3-370ed29d9549-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.986532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.986382 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:57.987339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.987310 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:08:57.987497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.987477 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "kube-api-access-2zk2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:57.987688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.987674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:08:57.987900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.987888 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy" pod "3251f60e-47f1-4fa7-abe3-370ed29d9549" (UID: "3251f60e-47f1-4fa7-abe3-370ed29d9549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:08:57.989772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:57.989751 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:08:58.026837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.026813 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:08:58.088225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.088171 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:58.088225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.088193 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zk2n\" (UniqueName: \"kubernetes.io/projected/3251f60e-47f1-4fa7-abe3-370ed29d9549-kube-api-access-2zk2n\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:58.088225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.088203 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:58.088225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.088212 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3251f60e-47f1-4fa7-abe3-370ed29d9549-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:08:58.628042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.628010 4543 generic.go:356] "Generic (PLEG): container finished" podID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerID="2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494" exitCode=0 Oct 29 22:08:58.628320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.628066 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" Oct 29 22:08:58.628320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.628069 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerDied","Data":"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494"} Oct 29 22:08:58.628320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.628146 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq" event={"ID":"3251f60e-47f1-4fa7-abe3-370ed29d9549","Type":"ContainerDied","Data":"320ac2ad1b9680880852d6e2ba13f2d9f54463f3f290c8edc90ee88cd0a84deb"} Oct 29 22:08:58.628320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.628170 4543 scope.go:122] "RemoveContainer" containerID="2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494" Oct 29 22:08:58.639124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.639104 4543 scope.go:122] "RemoveContainer" containerID="11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3" Oct 29 22:08:58.651854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.651831 4543 scope.go:122] "RemoveContainer" containerID="2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494" Oct 29 22:08:58.652334 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:58.652313 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494\": container with ID starting with 2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494 not found: ID does not exist" containerID="2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494" Oct 29 22:08:58.652430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.652341 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494"} err="failed to get container status \"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494\": rpc error: code = NotFound desc = could not find container \"2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494\": container with ID starting with 2040d3ed12d102b1e424b45be3d6f337f7271315461d174bd2e2eb52e4793494 not found: ID does not exist" Oct 29 22:08:58.652430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.652357 4543 scope.go:122] "RemoveContainer" containerID="11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3" Oct 29 22:08:58.652644 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:08:58.652621 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3\": container with ID starting with 11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3 not found: ID does not exist" containerID="11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3" Oct 29 22:08:58.652703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.652654 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3"} err="failed to get container status \"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3\": rpc error: code = NotFound desc = could not find container \"11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3\": container with ID starting with 11526f21add4d150ff1ab0ea336e884ebf7410fa036d428a59ab199189ff70a3 not found: ID does not exist" Oct 29 22:08:58.689399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.689345 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:08:58.701936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:08:58.701902 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5bc531-openshift-default-5b77fb455-79jpq"] Oct 29 22:09:00.386556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:00.386515 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" path="/var/lib/kubelet/pods/3251f60e-47f1-4fa7-abe3-370ed29d9549/volumes" Oct 29 22:09:02.990450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:02.990388 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" podUID="1ec94360-4262-4f8e-996b-6019f6747a6e" containerName="payload-processing" containerID="cri-o://7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07" gracePeriod=25 Oct 29 22:09:03.027865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.027810 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" podUID="39c04a79-34d1-4e9f-8d6e-059447089222" containerName="payload-pre-processing" containerID="cri-o://b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c" gracePeriod=25 Oct 29 22:09:03.327404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.327370 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:09:03.375216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.375188 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:09:03.465216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465185 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\" (UniqueName: \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\") pod \"39c04a79-34d1-4e9f-8d6e-059447089222\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " Oct 29 22:09:03.465368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465316 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\" (UniqueName: \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\") pod \"1ec94360-4262-4f8e-996b-6019f6747a6e\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " Oct 29 22:09:03.465368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465346 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\") pod \"39c04a79-34d1-4e9f-8d6e-059447089222\" (UID: \"39c04a79-34d1-4e9f-8d6e-059447089222\") " Oct 29 22:09:03.465447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465420 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\") pod \"1ec94360-4262-4f8e-996b-6019f6747a6e\" (UID: \"1ec94360-4262-4f8e-996b-6019f6747a6e\") " Oct 29 22:09:03.465824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465793 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume" pod "39c04a79-34d1-4e9f-8d6e-059447089222" (UID: "39c04a79-34d1-4e9f-8d6e-059447089222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:03.465824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.465798 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume" pod "1ec94360-4262-4f8e-996b-6019f6747a6e" (UID: "1ec94360-4262-4f8e-996b-6019f6747a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:03.466048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.466024 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c04a79-34d1-4e9f-8d6e-059447089222-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:03.466048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.466044 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec94360-4262-4f8e-996b-6019f6747a6e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:03.467602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.467583 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7" pod "39c04a79-34d1-4e9f-8d6e-059447089222" (UID: "39c04a79-34d1-4e9f-8d6e-059447089222"). InnerVolumeSpecName "kube-api-access-2s6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:03.467705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.467611 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj" pod "1ec94360-4262-4f8e-996b-6019f6747a6e" (UID: "1ec94360-4262-4f8e-996b-6019f6747a6e"). InnerVolumeSpecName "kube-api-access-5wzcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:03.567669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.567613 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6g7\" (UniqueName: \"kubernetes.io/projected/39c04a79-34d1-4e9f-8d6e-059447089222-kube-api-access-2s6g7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:03.567669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.567636 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzcj\" (UniqueName: \"kubernetes.io/projected/1ec94360-4262-4f8e-996b-6019f6747a6e-kube-api-access-5wzcj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:03.598707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.598686 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:09:03.636531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.636505 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:09:03.667694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.667667 4543 generic.go:356] "Generic (PLEG): container finished" podID="1ec94360-4262-4f8e-996b-6019f6747a6e" containerID="7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07" exitCode=0 Oct 29 22:09:03.667812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.667708 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" Oct 29 22:09:03.667812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.667755 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" event={"ID":"1ec94360-4262-4f8e-996b-6019f6747a6e","Type":"ContainerDied","Data":"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07"} Oct 29 22:09:03.667812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.667804 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl" event={"ID":"1ec94360-4262-4f8e-996b-6019f6747a6e","Type":"ContainerDied","Data":"774bdca1e2916a5110657f23f2e6c45da0b032a8130ff704ec47b463396758d1"} Oct 29 22:09:03.667965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.667831 4543 scope.go:122] "RemoveContainer" containerID="7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07" Oct 29 22:09:03.669710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.669684 4543 generic.go:356] "Generic (PLEG): container finished" podID="39c04a79-34d1-4e9f-8d6e-059447089222" containerID="b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c" exitCode=0 Oct 29 22:09:03.669864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.669726 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" Oct 29 22:09:03.669864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.669747 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" event={"ID":"39c04a79-34d1-4e9f-8d6e-059447089222","Type":"ContainerDied","Data":"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c"} Oct 29 22:09:03.669864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.669800 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l" event={"ID":"39c04a79-34d1-4e9f-8d6e-059447089222","Type":"ContainerDied","Data":"c29143af2fb4ee107e6ec80c68bfa5c8a1110defb9a533da98506e272eea4dc2"} Oct 29 22:09:03.678880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.678859 4543 scope.go:122] "RemoveContainer" containerID="7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07" Oct 29 22:09:03.679183 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:03.679166 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07\": container with ID starting with 7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07 not found: ID does not exist" containerID="7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07" Oct 29 22:09:03.679280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.679195 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07"} err="failed to get container status \"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07\": rpc error: code = NotFound desc = could not find container \"7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07\": container with ID starting with 7ccf31e5ce9a3c71becbd45ae569450593dcb960317edf03dc7497510b48ad07 not found: ID does not exist" Oct 29 22:09:03.679280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.679219 4543 scope.go:122] "RemoveContainer" containerID="b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c" Oct 29 22:09:03.692544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.692526 4543 scope.go:122] "RemoveContainer" containerID="b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c" Oct 29 22:09:03.693134 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:03.692989 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c\": container with ID starting with b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c not found: ID does not exist" containerID="b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c" Oct 29 22:09:03.693134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.693030 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c"} err="failed to get container status \"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c\": rpc error: code = NotFound desc = could not find container \"b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c\": container with ID starting with b1c5432d27e9b6bfcb5c60fcafbb5dd6cdfa894f732eb44b9eb8a173bdb9435c not found: ID does not exist" Oct 29 22:09:03.750000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.749940 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:09:03.753803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.753778 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d9498-77cb8787b4-bgndl"] Oct 29 22:09:03.790816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.790788 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:09:03.795265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:03.795242 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d9498-7d6f74c9b7-sgg2l"] Oct 29 22:09:04.417599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.417514 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec94360-4262-4f8e-996b-6019f6747a6e" path="/var/lib/kubelet/pods/1ec94360-4262-4f8e-996b-6019f6747a6e/volumes" Oct 29 22:09:04.418925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.418893 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c04a79-34d1-4e9f-8d6e-059447089222" path="/var/lib/kubelet/pods/39c04a79-34d1-4e9f-8d6e-059447089222/volumes" Oct 29 22:09:04.993427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.993391 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:09:04.994421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994403 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994423 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994435 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39c04a79-34d1-4e9f-8d6e-059447089222" containerName="payload-pre-processing" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994440 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c04a79-34d1-4e9f-8d6e-059447089222" containerName="payload-pre-processing" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994466 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d2c79b9f-2563-46d1-ad53-c9a07337826a" containerName="revoke-keys" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994471 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c79b9f-2563-46d1-ad53-c9a07337826a" containerName="revoke-keys" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994485 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994490 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994501 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1ec94360-4262-4f8e-996b-6019f6747a6e" containerName="payload-processing" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994506 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec94360-4262-4f8e-996b-6019f6747a6e" containerName="payload-processing" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994595 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c79b9f-2563-46d1-ad53-c9a07337826a" containerName="revoke-keys" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994605 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994611 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec94360-4262-4f8e-996b-6019f6747a6e" containerName="payload-processing" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994623 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3251f60e-47f1-4fa7-abe3-370ed29d9549" containerName="istio-proxy" Oct 29 22:09:04.994713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.994629 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c04a79-34d1-4e9f-8d6e-059447089222" containerName="payload-pre-processing" Oct 29 22:09:04.998231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:04.998210 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.015757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.015731 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:09:05.025780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.025401 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.029700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.029670 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:09:05.050394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.050367 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:09:05.095599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095574 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.095745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095629 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.095745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095719 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.095842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7r9\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.095881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095846 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.095970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.095950 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.096038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096172 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096307 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096356 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.096471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096426 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096480 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096606 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096674 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.096735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096718 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.096785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096751 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.096838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096827 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.096876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.096862 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh5t\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.197924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.197895 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.197948 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.197973 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7r9\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198004 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198089 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198127 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198191 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198226 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198267 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198294 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198340 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198405 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198435 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198459 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198479 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198494 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.198567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198546 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198589 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198626 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh5t\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198682 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198695 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.199241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.198995 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.199586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.199338 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.199376 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.199591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.199788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.199750 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.200382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.200358 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.200929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.200906 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.201252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.201232 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.201816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.201792 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.201894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.201801 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.224758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.224731 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.225907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.225882 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7r9\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.226179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.226161 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh5t\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.226235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.226190 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.321177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.321058 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:05.365026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.364962 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:05.679343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.679275 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:09:05.693756 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:05.693729 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811345cd_c9e3_4159_a083_6f2749461f3c.slice/crio-5edcf4b0b6087c07bb24e8302de74304a367168c5d09295e78e82a19a9f72f7d WatchSource:0}: Error finding container 5edcf4b0b6087c07bb24e8302de74304a367168c5d09295e78e82a19a9f72f7d: Status 404 returned error can't find the container with id 5edcf4b0b6087c07bb24e8302de74304a367168c5d09295e78e82a19a9f72f7d Oct 29 22:09:05.696226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.696193 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:05.696297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.696283 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:05.696342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.696319 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:05.725256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.725230 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:09:05.725348 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:05.725304 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7a6a14_5efa_4717_b459_fa772fb664aa.slice/crio-49ff22db7209a7ebb1e9a8a610d922d2ece1a8049601a83147fabe3df2861473 WatchSource:0}: Error finding container 49ff22db7209a7ebb1e9a8a610d922d2ece1a8049601a83147fabe3df2861473: Status 404 returned error can't find the container with id 49ff22db7209a7ebb1e9a8a610d922d2ece1a8049601a83147fabe3df2861473 Oct 29 22:09:05.728510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.728471 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:05.728681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.728563 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:05.728681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:05.728609 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:06.704602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.704558 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" event={"ID":"2f7a6a14-5efa-4717-b459-fa772fb664aa","Type":"ContainerStarted","Data":"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a"} Oct 29 22:09:06.705109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.704625 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" event={"ID":"2f7a6a14-5efa-4717-b459-fa772fb664aa","Type":"ContainerStarted","Data":"49ff22db7209a7ebb1e9a8a610d922d2ece1a8049601a83147fabe3df2861473"} Oct 29 22:09:06.706452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.706426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" event={"ID":"811345cd-c9e3-4159-a083-6f2749461f3c","Type":"ContainerStarted","Data":"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72"} Oct 29 22:09:06.706581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.706479 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" event={"ID":"811345cd-c9e3-4159-a083-6f2749461f3c","Type":"ContainerStarted","Data":"5edcf4b0b6087c07bb24e8302de74304a367168c5d09295e78e82a19a9f72f7d"} Oct 29 22:09:06.782136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.780450 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podStartSLOduration=2.780431856 podStartE2EDuration="2.780431856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.779143979 +0000 UTC m=+1608.521074652" watchObservedRunningTime="2026-10-29 22:09:06.780431856 +0000 UTC m=+1608.522362529" Oct 29 22:09:06.854162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:06.845386 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podStartSLOduration=2.8453629879999998 podStartE2EDuration="2.845362988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.826815694 +0000 UTC m=+1608.568746396" watchObservedRunningTime="2026-10-29 22:09:06.845362988 +0000 UTC m=+1608.587293661" Oct 29 22:09:07.321715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.321679 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:07.323618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.323591 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:07.323758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.323640 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:07.379656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.379626 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:07.380697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.380665 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:07.380849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:07.380831 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:08.116679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.116648 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t"] Oct 29 22:09:08.120298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.120279 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.192264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.191959 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t"] Oct 29 22:09:08.203793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.195708 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:09:08.203793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.195850 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:09:08.203793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.195947 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:09:08.203793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.196010 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:09:08.299040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.299003 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wlhr\" (UniqueName: \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.299225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.299067 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.299309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.299286 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.321799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.321772 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:08.321935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.321824 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:08.376801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.376725 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:08.377049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.377022 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:08.400558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.400523 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.400728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.400711 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlhr\" (UniqueName: \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.400787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.400766 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.401481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.401454 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.403707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.403683 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.445770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.445743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlhr\" (UniqueName: \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\") pod \"maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.504366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.504340 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:08.601127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.600663 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" podUID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" containerName="payload-processing" containerID="cri-o://f464780fb6a4a8ffe06ce9275d2a315bb9771c1688a77c12bf19811a79086516" gracePeriod=25 Oct 29 22:09:08.651199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.638176 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" podUID="e7e06426-c97d-47c2-b662-d3c235dd9706" containerName="payload-pre-processing" containerID="cri-o://2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6" gracePeriod=25 Oct 29 22:09:08.726033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.726004 4543 generic.go:356] "Generic (PLEG): container finished" podID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" containerID="f464780fb6a4a8ffe06ce9275d2a315bb9771c1688a77c12bf19811a79086516" exitCode=0 Oct 29 22:09:08.726898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.726877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" event={"ID":"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548","Type":"ContainerDied","Data":"f464780fb6a4a8ffe06ce9275d2a315bb9771c1688a77c12bf19811a79086516"} Oct 29 22:09:08.897303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:08.896889 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t"] Oct 29 22:09:08.910949 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:08.910903 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93835c95_388e_4404_8827_b2a1713c9a90.slice/crio-06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a WatchSource:0}: Error finding container 06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a: Status 404 returned error can't find the container with id 06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a Oct 29 22:09:09.227630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.227602 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:09:09.324011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.321861 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:09.324011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.321924 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:09.376099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.375114 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\") pod \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " Oct 29 22:09:09.376099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.375177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\" (UniqueName: \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\") pod \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\" (UID: \"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548\") " Oct 29 22:09:09.383368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.377614 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume" pod "8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" (UID: "8f7a915d-a0f1-4d49-b2fe-6e32efaaf548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:09.383368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.381211 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:09.383368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.381259 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:09.385825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.385796 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9" pod "8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" (UID: "8f7a915d-a0f1-4d49-b2fe-6e32efaaf548"). InnerVolumeSpecName "kube-api-access-7pvq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:09.396300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.396281 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:09:09.487162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.485019 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\" (UniqueName: \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\") pod \"e7e06426-c97d-47c2-b662-d3c235dd9706\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " Oct 29 22:09:09.487162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.485409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\") pod \"e7e06426-c97d-47c2-b662-d3c235dd9706\" (UID: \"e7e06426-c97d-47c2-b662-d3c235dd9706\") " Oct 29 22:09:09.487162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.486062 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:09.487162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.486100 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvq9\" (UniqueName: \"kubernetes.io/projected/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548-kube-api-access-7pvq9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:09.487162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.486981 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume" pod "e7e06426-c97d-47c2-b662-d3c235dd9706" (UID: "e7e06426-c97d-47c2-b662-d3c235dd9706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:09.492617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.490311 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t" pod "e7e06426-c97d-47c2-b662-d3c235dd9706" (UID: "e7e06426-c97d-47c2-b662-d3c235dd9706"). InnerVolumeSpecName "kube-api-access-rf77t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:09.587379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.587343 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf77t\" (UniqueName: \"kubernetes.io/projected/e7e06426-c97d-47c2-b662-d3c235dd9706-kube-api-access-rf77t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:09.587379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.587379 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e06426-c97d-47c2-b662-d3c235dd9706-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:09.782171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.781888 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" event={"ID":"93835c95-388e-4404-8827-b2a1713c9a90","Type":"ContainerStarted","Data":"0f566973eb8fc391de5a594a5281390e0d0b08c2dc6225ae9afafa8545a4ce8c"} Oct 29 22:09:09.782171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.781977 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" event={"ID":"93835c95-388e-4404-8827-b2a1713c9a90","Type":"ContainerStarted","Data":"06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a"} Oct 29 22:09:09.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.785435 4543 generic.go:356] "Generic (PLEG): container finished" podID="e7e06426-c97d-47c2-b662-d3c235dd9706" containerID="2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6" exitCode=0 Oct 29 22:09:09.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.785624 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" event={"ID":"e7e06426-c97d-47c2-b662-d3c235dd9706","Type":"ContainerDied","Data":"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6"} Oct 29 22:09:09.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.785688 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" event={"ID":"e7e06426-c97d-47c2-b662-d3c235dd9706","Type":"ContainerDied","Data":"2ba9cb1927a9fcfb5186f3191fecfcc160b0cda7e0c8864409ec1e7f67c73711"} Oct 29 22:09:09.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.785738 4543 scope.go:122] "RemoveContainer" containerID="2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6" Oct 29 22:09:09.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.785916 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj" Oct 29 22:09:09.814103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.813723 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" Oct 29 22:09:09.814103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.813745 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb" event={"ID":"8f7a915d-a0f1-4d49-b2fe-6e32efaaf548","Type":"ContainerDied","Data":"4bbd13dbbbebc512c63e2de6191aae913edf712b127624486abb9e910d1ee215"} Oct 29 22:09:09.826238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.826198 4543 scope.go:122] "RemoveContainer" containerID="2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6" Oct 29 22:09:09.826511 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:09.826489 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6\": container with ID starting with 2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6 not found: ID does not exist" containerID="2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6" Oct 29 22:09:09.826581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.826517 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6"} err="failed to get container status \"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6\": rpc error: code = NotFound desc = could not find container \"2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6\": container with ID starting with 2ad0548727ed81ff224dfcf0c5138b2859360b3c4b27ccfcd79a3422ab2d11c6 not found: ID does not exist" Oct 29 22:09:09.826581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.826533 4543 scope.go:122] "RemoveContainer" containerID="f464780fb6a4a8ffe06ce9275d2a315bb9771c1688a77c12bf19811a79086516" Oct 29 22:09:09.887532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.886888 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" podStartSLOduration=1.8868692569999999 podStartE2EDuration="1.886869257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:09.842622251 +0000 UTC m=+1611.584552929" watchObservedRunningTime="2026-10-29 22:09:09.886869257 +0000 UTC m=+1611.628799944" Oct 29 22:09:09.893720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.893696 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:09:09.896884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.896858 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6153f8-57885cbc79-4rfgj"] Oct 29 22:09:09.980103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.977870 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:09:09.986861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:09.986831 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6153f8-569b5b94c4-wntsb"] Oct 29 22:09:10.321680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.321647 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:10.322056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.321698 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:10.373807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.373406 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:10.373807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.373460 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:10.390553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.390524 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" path="/var/lib/kubelet/pods/8f7a915d-a0f1-4d49-b2fe-6e32efaaf548/volumes" Oct 29 22:09:10.390994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.390979 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e06426-c97d-47c2-b662-d3c235dd9706" path="/var/lib/kubelet/pods/e7e06426-c97d-47c2-b662-d3c235dd9706/volumes" Oct 29 22:09:10.826876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.826843 4543 generic.go:356] "Generic (PLEG): container finished" podID="93835c95-388e-4404-8827-b2a1713c9a90" containerID="0f566973eb8fc391de5a594a5281390e0d0b08c2dc6225ae9afafa8545a4ce8c" exitCode=0 Oct 29 22:09:10.827039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:10.826944 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" event={"ID":"93835c95-388e-4404-8827-b2a1713c9a90","Type":"ContainerDied","Data":"0f566973eb8fc391de5a594a5281390e0d0b08c2dc6225ae9afafa8545a4ce8c"} Oct 29 22:09:11.322258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:11.322228 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:11.322639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:11.322279 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:11.377098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:11.376642 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:11.377098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:11.376701 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:12.162978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.162913 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:12.192049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.192016 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\" (UniqueName: \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\") pod \"93835c95-388e-4404-8827-b2a1713c9a90\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " Oct 29 22:09:12.192287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.192270 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\") pod \"93835c95-388e-4404-8827-b2a1713c9a90\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " Oct 29 22:09:12.192373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.192353 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\") pod \"93835c95-388e-4404-8827-b2a1713c9a90\" (UID: \"93835c95-388e-4404-8827-b2a1713c9a90\") " Oct 29 22:09:12.192771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.192727 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca" pod "93835c95-388e-4404-8827-b2a1713c9a90" (UID: "93835c95-388e-4404-8827-b2a1713c9a90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:12.193117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.193093 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:12.194415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.194393 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr" pod "93835c95-388e-4404-8827-b2a1713c9a90" (UID: "93835c95-388e-4404-8827-b2a1713c9a90"). InnerVolumeSpecName "kube-api-access-8wlhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:12.194900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.194881 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca" pod "93835c95-388e-4404-8827-b2a1713c9a90" (UID: "93835c95-388e-4404-8827-b2a1713c9a90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:12.294716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.294682 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93835c95-388e-4404-8827-b2a1713c9a90-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:12.294716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.294713 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wlhr\" (UniqueName: \"kubernetes.io/projected/93835c95-388e-4404-8827-b2a1713c9a90-kube-api-access-8wlhr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:12.321706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.321666 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:09:12.321865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.321713 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 29 22:09:12.366345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.366312 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:09:12.366769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.366355 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 22:09:12.892091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.892040 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" Oct 29 22:09:12.892091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.892058 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t" event={"ID":"93835c95-388e-4404-8827-b2a1713c9a90","Type":"ContainerDied","Data":"06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a"} Oct 29 22:09:12.892305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:12.892134 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06fa9ad4364411060c7d72802c809f53f8b84e6b04753e598082da2375d5af2a" Oct 29 22:09:13.744903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.744855 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:13.745419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.745341 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:13.746759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.746738 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:09:13.817610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.817574 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:13.818526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.818499 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:13.819293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:13.819270 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:09:18.308252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:18.308217 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2"] Oct 29 22:09:18.339275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:18.339239 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d9498-pvzx2"] Oct 29 22:09:18.388801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:18.388774 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babf5345-6434-484f-b4e8-100ebf55a2cb" path="/var/lib/kubelet/pods/babf5345-6434-484f-b4e8-100ebf55a2cb/volumes" Oct 29 22:09:20.765702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:20.765402 4543 scope.go:122] "RemoveContainer" containerID="f3b64c4e25e8e4e04cd18a375091487d60b150be58d9c4e67f4e2d1acaf7a68e" Oct 29 22:09:20.778809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:20.778760 4543 scope.go:122] "RemoveContainer" containerID="c3ca949f0a1549dcf224f02f2fe550c19fae2e188accf63a7b0343ef12012640" Oct 29 22:09:25.649737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:25.649692 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:09:25.650273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:25.649986 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" containerID="cri-o://0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c" gracePeriod=30 Oct 29 22:09:27.024030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.023998 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh"] Oct 29 22:09:27.024725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024708 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" containerName="payload-processing" Oct 29 22:09:27.024772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024727 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" containerName="payload-processing" Oct 29 22:09:27.024772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024754 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e7e06426-c97d-47c2-b662-d3c235dd9706" containerName="payload-pre-processing" Oct 29 22:09:27.024772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024759 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e06426-c97d-47c2-b662-d3c235dd9706" containerName="payload-pre-processing" Oct 29 22:09:27.024772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024770 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93835c95-388e-4404-8827-b2a1713c9a90" containerName="revoke-keys" Oct 29 22:09:27.024913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024775 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93835c95-388e-4404-8827-b2a1713c9a90" containerName="revoke-keys" Oct 29 22:09:27.024913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024893 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e06426-c97d-47c2-b662-d3c235dd9706" containerName="payload-pre-processing" Oct 29 22:09:27.024913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024901 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93835c95-388e-4404-8827-b2a1713c9a90" containerName="revoke-keys" Oct 29 22:09:27.025020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.024916 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7a915d-a0f1-4d49-b2fe-6e32efaaf548" containerName="payload-processing" Oct 29 22:09:27.029256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.029238 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.048128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.048096 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh"] Oct 29 22:09:27.057692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.057640 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:09:27.057692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.057687 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:09:27.057863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.057685 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:09:27.061630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.061613 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:09:27.117670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.117642 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.117821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.117677 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgshb\" (UniqueName: \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.117821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.117708 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.219274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.219249 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.219408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.219279 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgshb\" (UniqueName: \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.219408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.219306 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.219873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.219855 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.221789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.221766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.244790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.244770 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgshb\" (UniqueName: \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\") pod \"maas-api-revoke-sub-e897b851-vgxzh\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.342700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.342652 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:27.548417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.548390 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn"] Oct 29 22:09:27.552434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.552416 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.573475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.573452 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn"] Oct 29 22:09:27.624049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.623992 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.624049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.624036 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtpz\" (UniqueName: \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.624224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.624059 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.654837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.654807 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh"] Oct 29 22:09:27.656130 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:27.656102 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758943d0_7f41_43d5_ab7f_e45b39087124.slice/crio-19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0 WatchSource:0}: Error finding container 19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0: Status 404 returned error can't find the container with id 19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0 Oct 29 22:09:27.725207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.725183 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.725305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.725225 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtpz\" (UniqueName: \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.725305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.725252 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.725909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.725884 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.727449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.727432 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.750560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.750539 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtpz\" (UniqueName: \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\") pod \"maas-api-revoke-sub-57326934-pnpbn\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:27.866703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:27.866677 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:28.029786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:28.029749 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" event={"ID":"758943d0-7f41-43d5-ab7f-e45b39087124","Type":"ContainerStarted","Data":"19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0"} Oct 29 22:09:28.051837 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:28.051810 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:09:28.178231 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:28.178200 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d8003a_89e1_45dc_891b_f2dff5b62035.slice/crio-920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42 WatchSource:0}: Error finding container 920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42: Status 404 returned error can't find the container with id 920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42 Oct 29 22:09:28.179056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:28.179023 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn"] Oct 29 22:09:29.039358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:29.039321 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" event={"ID":"c6d8003a-89e1-45dc-891b-f2dff5b62035","Type":"ContainerStarted","Data":"2fa7fda13d268f9f773ccfdc86f1ce767d0e82c5635d2fd4ef92b90be7fb0e66"} Oct 29 22:09:29.039794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:29.039386 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" event={"ID":"c6d8003a-89e1-45dc-891b-f2dff5b62035","Type":"ContainerStarted","Data":"920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42"} Oct 29 22:09:29.041123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:29.041101 4543 generic.go:356] "Generic (PLEG): container finished" podID="758943d0-7f41-43d5-ab7f-e45b39087124" containerID="b173402577f461b7c659636ab54b6ecf7ebe910ad57e8ea4acaae7ce5b84cb4e" exitCode=0 Oct 29 22:09:29.041246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:29.041141 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" event={"ID":"758943d0-7f41-43d5-ab7f-e45b39087124","Type":"ContainerDied","Data":"b173402577f461b7c659636ab54b6ecf7ebe910ad57e8ea4acaae7ce5b84cb4e"} Oct 29 22:09:29.085441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:29.085393 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" podStartSLOduration=2.085377137 podStartE2EDuration="2.085377137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:29.082549456 +0000 UTC m=+1630.824480130" watchObservedRunningTime="2026-10-29 22:09:29.085377137 +0000 UTC m=+1630.827307811" Oct 29 22:09:30.050419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.050386 4543 generic.go:356] "Generic (PLEG): container finished" podID="c6d8003a-89e1-45dc-891b-f2dff5b62035" containerID="2fa7fda13d268f9f773ccfdc86f1ce767d0e82c5635d2fd4ef92b90be7fb0e66" exitCode=0 Oct 29 22:09:30.050840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.050473 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" event={"ID":"c6d8003a-89e1-45dc-891b-f2dff5b62035","Type":"ContainerDied","Data":"2fa7fda13d268f9f773ccfdc86f1ce767d0e82c5635d2fd4ef92b90be7fb0e66"} Oct 29 22:09:30.241826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.241803 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:30.357192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.357126 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\" (UniqueName: \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\") pod \"758943d0-7f41-43d5-ab7f-e45b39087124\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " Oct 29 22:09:30.357422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.357407 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\") pod \"758943d0-7f41-43d5-ab7f-e45b39087124\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " Oct 29 22:09:30.357471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.357435 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\") pod \"758943d0-7f41-43d5-ab7f-e45b39087124\" (UID: \"758943d0-7f41-43d5-ab7f-e45b39087124\") " Oct 29 22:09:30.357866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.357838 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca" pod "758943d0-7f41-43d5-ab7f-e45b39087124" (UID: "758943d0-7f41-43d5-ab7f-e45b39087124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:30.358181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.357955 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:30.359312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.359286 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb" pod "758943d0-7f41-43d5-ab7f-e45b39087124" (UID: "758943d0-7f41-43d5-ab7f-e45b39087124"). InnerVolumeSpecName "kube-api-access-mgshb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:30.359595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.359579 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca" pod "758943d0-7f41-43d5-ab7f-e45b39087124" (UID: "758943d0-7f41-43d5-ab7f-e45b39087124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:30.459217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.459185 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758943d0-7f41-43d5-ab7f-e45b39087124-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:30.459217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.459212 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgshb\" (UniqueName: \"kubernetes.io/projected/758943d0-7f41-43d5-ab7f-e45b39087124-kube-api-access-mgshb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:30.911543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:30.911517 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:09:31.060061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.060029 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" Oct 29 22:09:31.060526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.060031 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh" event={"ID":"758943d0-7f41-43d5-ab7f-e45b39087124","Type":"ContainerDied","Data":"19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0"} Oct 29 22:09:31.060526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.060155 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19dcd4806fd382fbff6bb3caa8b31c651c1b2cef66e8ceddc42e28539ecadcb0" Oct 29 22:09:31.061995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.061971 4543 generic.go:356] "Generic (PLEG): container finished" podID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerID="0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c" exitCode=0 Oct 29 22:09:31.062164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.062017 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" Oct 29 22:09:31.062254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.062228 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerDied","Data":"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c"} Oct 29 22:09:31.062298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.062285 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt" event={"ID":"fcd1b6f9-c443-4c8f-8498-febb7813179c","Type":"ContainerDied","Data":"249714c4577f02dea616e2e87d8e5d687141d1fdef59635084df8e04cc6ffe52"} Oct 29 22:09:31.062340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.062316 4543 scope.go:122] "RemoveContainer" containerID="0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c" Oct 29 22:09:31.067868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.067847 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.067961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.067939 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.067964 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068008 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068115 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:31.068308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068290 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068318 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:31.068436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068374 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068457 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\") pod \"fcd1b6f9-c443-4c8f-8498-febb7813179c\" (UID: \"fcd1b6f9-c443-4c8f-8498-febb7813179c\") " Oct 29 22:09:31.068556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068535 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:31.068725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.068707 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:31.069341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.069313 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.069341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.069347 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.069341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.069363 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.069341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.069375 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.070398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.070375 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:09:31.070781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.070759 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:31.071370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.071348 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:31.071483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.071409 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:31.071483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.071424 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn" pod "fcd1b6f9-c443-4c8f-8498-febb7813179c" (UID: "fcd1b6f9-c443-4c8f-8498-febb7813179c"). InnerVolumeSpecName "kube-api-access-xlskn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:31.074467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.074446 4543 scope.go:122] "RemoveContainer" containerID="de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7" Oct 29 22:09:31.085488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.085472 4543 scope.go:122] "RemoveContainer" containerID="0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c" Oct 29 22:09:31.085793 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:31.085774 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c\": container with ID starting with 0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c not found: ID does not exist" containerID="0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c" Oct 29 22:09:31.085878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.085799 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c"} err="failed to get container status \"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c\": rpc error: code = NotFound desc = could not find container \"0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c\": container with ID starting with 0c7a1636dfd7c5f8f5713ca03677e4aa564d535601c58e49b4be04998b9a5f4c not found: ID does not exist" Oct 29 22:09:31.085878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.085815 4543 scope.go:122] "RemoveContainer" containerID="de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7" Oct 29 22:09:31.086464 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:31.086447 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7\": container with ID starting with de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7 not found: ID does not exist" containerID="de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7" Oct 29 22:09:31.086546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.086467 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7"} err="failed to get container status \"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7\": rpc error: code = NotFound desc = could not find container \"de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7\": container with ID starting with de0e87e7d21b01a5441f39a10c523498e7910b3554146cad012efecaf47635d7 not found: ID does not exist" Oct 29 22:09:31.171205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.171171 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.171205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.171208 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.171443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.171221 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd1b6f9-c443-4c8f-8498-febb7813179c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.171443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.171235 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlskn\" (UniqueName: \"kubernetes.io/projected/fcd1b6f9-c443-4c8f-8498-febb7813179c-kube-api-access-xlskn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.171443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.171248 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd1b6f9-c443-4c8f-8498-febb7813179c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.208467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.208444 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:31.373832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.373806 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\") pod \"c6d8003a-89e1-45dc-891b-f2dff5b62035\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " Oct 29 22:09:31.373966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.373954 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\" (UniqueName: \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\") pod \"c6d8003a-89e1-45dc-891b-f2dff5b62035\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " Oct 29 22:09:31.374029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.374014 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\") pod \"c6d8003a-89e1-45dc-891b-f2dff5b62035\" (UID: \"c6d8003a-89e1-45dc-891b-f2dff5b62035\") " Oct 29 22:09:31.374355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.374330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca" pod "c6d8003a-89e1-45dc-891b-f2dff5b62035" (UID: "c6d8003a-89e1-45dc-891b-f2dff5b62035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:31.374488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.374465 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.375940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.375920 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca" pod "c6d8003a-89e1-45dc-891b-f2dff5b62035" (UID: "c6d8003a-89e1-45dc-891b-f2dff5b62035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:31.376223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.376193 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz" pod "c6d8003a-89e1-45dc-891b-f2dff5b62035" (UID: "c6d8003a-89e1-45dc-891b-f2dff5b62035"). InnerVolumeSpecName "kube-api-access-fmtpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:31.414688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.414668 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:09:31.418104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.418062 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d9498-openshift-default-799b659457-xcdkt"] Oct 29 22:09:31.475970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.475948 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8003a-89e1-45dc-891b-f2dff5b62035-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:31.475970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:31.475969 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmtpz\" (UniqueName: \"kubernetes.io/projected/c6d8003a-89e1-45dc-891b-f2dff5b62035-kube-api-access-fmtpz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:32.073225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:32.073190 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" Oct 29 22:09:32.073225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:32.073212 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn" event={"ID":"c6d8003a-89e1-45dc-891b-f2dff5b62035","Type":"ContainerDied","Data":"920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42"} Oct 29 22:09:32.073845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:32.073261 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920afcaaba191f41fd21ca28e57e77cf523fed5f1843abc24924169281ecbf42" Oct 29 22:09:32.386506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:32.386432 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" path="/var/lib/kubelet/pods/fcd1b6f9-c443-4c8f-8498-febb7813179c/volumes" Oct 29 22:09:37.982486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.982454 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp"] Oct 29 22:09:37.983131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983117 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c6d8003a-89e1-45dc-891b-f2dff5b62035" containerName="revoke-keys" Oct 29 22:09:37.983182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983132 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d8003a-89e1-45dc-891b-f2dff5b62035" containerName="revoke-keys" Oct 29 22:09:37.983182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983149 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:37.983182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983155 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:37.983182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983179 4543 cpu_manager.go:414] "containerMap: removing container" podUID="758943d0-7f41-43d5-ab7f-e45b39087124" containerName="revoke-keys" Oct 29 22:09:37.983314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983184 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758943d0-7f41-43d5-ab7f-e45b39087124" containerName="revoke-keys" Oct 29 22:09:37.983314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983303 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:37.983314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983312 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d8003a-89e1-45dc-891b-f2dff5b62035" containerName="revoke-keys" Oct 29 22:09:37.983411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983322 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:37.983411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.983332 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="758943d0-7f41-43d5-ab7f-e45b39087124" containerName="revoke-keys" Oct 29 22:09:37.986729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:37.986707 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.007061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.007038 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp"] Oct 29 22:09:38.014638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.014618 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:09:38.014880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.014860 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:09:38.079100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.079051 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:09:38.079215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.079134 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:09:38.152193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.152172 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbkv\" (UniqueName: \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.152324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.152303 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.152365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.152353 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.253938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.253875 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.254049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.254007 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbkv\" (UniqueName: \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.254147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.254107 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.254580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.254557 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.256310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.256290 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.279399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.279378 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbkv\" (UniqueName: \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\") pod \"maas-api-revoke-sub-58362f62-qcmjp\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.300043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.300025 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:38.609158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:38.609133 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp"] Oct 29 22:09:38.610269 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:38.610243 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf08d05c_7f5b_479f_b795_de095602ac80.slice/crio-bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903 WatchSource:0}: Error finding container bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903: Status 404 returned error can't find the container with id bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903 Oct 29 22:09:39.127146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.127111 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" event={"ID":"af08d05c-7f5b-479f-b795-de095602ac80","Type":"ContainerStarted","Data":"12d110cb8bdbbde2c7b67cf8c971a4a3d7344b57da594b024cfaf32fc756564b"} Oct 29 22:09:39.127516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.127174 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" event={"ID":"af08d05c-7f5b-479f-b795-de095602ac80","Type":"ContainerStarted","Data":"bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903"} Oct 29 22:09:39.175101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.175023 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" podStartSLOduration=2.175011088 podStartE2EDuration="2.175011088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:39.172339196 +0000 UTC m=+1640.914269870" watchObservedRunningTime="2026-10-29 22:09:39.175011088 +0000 UTC m=+1640.916941761" Oct 29 22:09:39.472297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.472273 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:09:39.472480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.472460 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" podUID="3650003a-e544-4cf4-a880-7ff2b8849d55" containerName="payload-processing" containerID="cri-o://277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471" gracePeriod=30 Oct 29 22:09:39.498913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.498885 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:09:39.499319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.499284 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" podUID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" containerName="payload-pre-processing" containerID="cri-o://af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7" gracePeriod=30 Oct 29 22:09:39.910886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.909773 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:09:39.923133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.922405 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\" (UniqueName: \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\") pod \"3650003a-e544-4cf4-a880-7ff2b8849d55\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " Oct 29 22:09:39.923133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.922483 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\") pod \"3650003a-e544-4cf4-a880-7ff2b8849d55\" (UID: \"3650003a-e544-4cf4-a880-7ff2b8849d55\") " Oct 29 22:09:39.931967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.923718 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume" pod "3650003a-e544-4cf4-a880-7ff2b8849d55" (UID: "3650003a-e544-4cf4-a880-7ff2b8849d55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:39.931967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.927392 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj" pod "3650003a-e544-4cf4-a880-7ff2b8849d55" (UID: "3650003a-e544-4cf4-a880-7ff2b8849d55"). InnerVolumeSpecName "kube-api-access-69dkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:39.976598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:39.976510 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:09:40.029969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.029932 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\" (UniqueName: \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\") pod \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " Oct 29 22:09:40.030205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.030185 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\") pod \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\" (UID: \"79034d70-eb2d-4f48-a4b2-d2b650ce6975\") " Oct 29 22:09:40.030787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.030759 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69dkj\" (UniqueName: \"kubernetes.io/projected/3650003a-e544-4cf4-a880-7ff2b8849d55-kube-api-access-69dkj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.030874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.030791 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3650003a-e544-4cf4-a880-7ff2b8849d55-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.031190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.031141 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume" pod "79034d70-eb2d-4f48-a4b2-d2b650ce6975" (UID: "79034d70-eb2d-4f48-a4b2-d2b650ce6975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:40.032784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.032760 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r" pod "79034d70-eb2d-4f48-a4b2-d2b650ce6975" (UID: "79034d70-eb2d-4f48-a4b2-d2b650ce6975"). InnerVolumeSpecName "kube-api-access-2xl4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:40.132158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.132120 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xl4r\" (UniqueName: \"kubernetes.io/projected/79034d70-eb2d-4f48-a4b2-d2b650ce6975-kube-api-access-2xl4r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.132158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.132155 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79034d70-eb2d-4f48-a4b2-d2b650ce6975-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.138795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.138769 4543 generic.go:356] "Generic (PLEG): container finished" podID="af08d05c-7f5b-479f-b795-de095602ac80" containerID="12d110cb8bdbbde2c7b67cf8c971a4a3d7344b57da594b024cfaf32fc756564b" exitCode=0 Oct 29 22:09:40.138948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.138853 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" event={"ID":"af08d05c-7f5b-479f-b795-de095602ac80","Type":"ContainerDied","Data":"12d110cb8bdbbde2c7b67cf8c971a4a3d7344b57da594b024cfaf32fc756564b"} Oct 29 22:09:40.140411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.140387 4543 generic.go:356] "Generic (PLEG): container finished" podID="3650003a-e544-4cf4-a880-7ff2b8849d55" containerID="277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471" exitCode=0 Oct 29 22:09:40.140552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.140426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" event={"ID":"3650003a-e544-4cf4-a880-7ff2b8849d55","Type":"ContainerDied","Data":"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471"} Oct 29 22:09:40.140552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.140474 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" event={"ID":"3650003a-e544-4cf4-a880-7ff2b8849d55","Type":"ContainerDied","Data":"14669e8189351d759dbd85cd533d584a1766919e1b056eba55fc1a90159574be"} Oct 29 22:09:40.140552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.140437 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l" Oct 29 22:09:40.140552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.140501 4543 scope.go:122] "RemoveContainer" containerID="277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471" Oct 29 22:09:40.141934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.141916 4543 generic.go:356] "Generic (PLEG): container finished" podID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" containerID="af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7" exitCode=0 Oct 29 22:09:40.142036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.141943 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" event={"ID":"79034d70-eb2d-4f48-a4b2-d2b650ce6975","Type":"ContainerDied","Data":"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7"} Oct 29 22:09:40.142036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.141972 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" Oct 29 22:09:40.142036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.141997 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc" event={"ID":"79034d70-eb2d-4f48-a4b2-d2b650ce6975","Type":"ContainerDied","Data":"eede619fff9be6e966f64f83c6fc4612d945e9453d18619ba384ee748fa89108"} Oct 29 22:09:40.157229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.157208 4543 scope.go:122] "RemoveContainer" containerID="277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471" Oct 29 22:09:40.157588 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:40.157561 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471\": container with ID starting with 277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471 not found: ID does not exist" containerID="277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471" Oct 29 22:09:40.157687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.157599 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471"} err="failed to get container status \"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471\": rpc error: code = NotFound desc = could not find container \"277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471\": container with ID starting with 277f0111bfa549cfe4a16226fc929cd804baacf415d187a62334c9e18ff1a471 not found: ID does not exist" Oct 29 22:09:40.157687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.157630 4543 scope.go:122] "RemoveContainer" containerID="af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7" Oct 29 22:09:40.171783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.171766 4543 scope.go:122] "RemoveContainer" containerID="af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7" Oct 29 22:09:40.172037 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:40.172016 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7\": container with ID starting with af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7 not found: ID does not exist" containerID="af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7" Oct 29 22:09:40.172105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.172046 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7"} err="failed to get container status \"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7\": rpc error: code = NotFound desc = could not find container \"af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7\": container with ID starting with af27840255fdfc12cc5171b02c493a502dfc04dbb5daea471d95ee1590efe2f7 not found: ID does not exist" Oct 29 22:09:40.227598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.227534 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:09:40.232995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.232967 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fd1d4e-6d4bc49c9d-2jj6l"] Oct 29 22:09:40.271122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.271101 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:09:40.278662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.278641 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fd1d4e-858768f8ff-lpjgc"] Oct 29 22:09:40.283630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.283611 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t"] Oct 29 22:09:40.287414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.287396 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fd1d4e-mbl9t"] Oct 29 22:09:40.333532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.333509 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52"] Oct 29 22:09:40.334241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334226 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:40.334292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334242 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd1b6f9-c443-4c8f-8498-febb7813179c" containerName="istio-proxy" Oct 29 22:09:40.334292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334261 4543 cpu_manager.go:414] "containerMap: removing container" podUID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" containerName="payload-pre-processing" Oct 29 22:09:40.334292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334270 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" containerName="payload-pre-processing" Oct 29 22:09:40.334292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334279 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3650003a-e544-4cf4-a880-7ff2b8849d55" containerName="payload-processing" Oct 29 22:09:40.334292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334284 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3650003a-e544-4cf4-a880-7ff2b8849d55" containerName="payload-processing" Oct 29 22:09:40.334459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334398 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" containerName="payload-pre-processing" Oct 29 22:09:40.334459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.334405 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3650003a-e544-4cf4-a880-7ff2b8849d55" containerName="payload-processing" Oct 29 22:09:40.337594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.337578 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.357570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.357549 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52"] Oct 29 22:09:40.399121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.399040 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3650003a-e544-4cf4-a880-7ff2b8849d55" path="/var/lib/kubelet/pods/3650003a-e544-4cf4-a880-7ff2b8849d55/volumes" Oct 29 22:09:40.409329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.399739 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79034d70-eb2d-4f48-a4b2-d2b650ce6975" path="/var/lib/kubelet/pods/79034d70-eb2d-4f48-a4b2-d2b650ce6975/volumes" Oct 29 22:09:40.409329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.400384 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93835c95-388e-4404-8827-b2a1713c9a90" path="/var/lib/kubelet/pods/93835c95-388e-4404-8827-b2a1713c9a90/volumes" Oct 29 22:09:40.437326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.437287 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.437694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.437668 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.437862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.437760 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvrj\" (UniqueName: \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.539505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.539423 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.539643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.539591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.539643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.539627 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvrj\" (UniqueName: \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.540172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.540153 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.542048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.542027 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.565520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.565494 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvrj\" (UniqueName: \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\") pod \"maas-api-revoke-sub-79e38bbc-rmp52\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.649921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.649898 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:40.957880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:40.957853 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52"] Oct 29 22:09:40.959004 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:40.958979 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6109265_a586_4622_90eb_2a75272e6a37.slice/crio-9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175 WatchSource:0}: Error finding container 9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175: Status 404 returned error can't find the container with id 9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175 Oct 29 22:09:41.153410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.153378 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" event={"ID":"c6109265-a586-4622-90eb-2a75272e6a37","Type":"ContainerStarted","Data":"9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175"} Oct 29 22:09:41.361105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.361044 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:41.451209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.451179 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\") pod \"af08d05c-7f5b-479f-b795-de095602ac80\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " Oct 29 22:09:41.451363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.451344 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\") pod \"af08d05c-7f5b-479f-b795-de095602ac80\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " Oct 29 22:09:41.451537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.451518 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\" (UniqueName: \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\") pod \"af08d05c-7f5b-479f-b795-de095602ac80\" (UID: \"af08d05c-7f5b-479f-b795-de095602ac80\") " Oct 29 22:09:41.451635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.451593 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca" pod "af08d05c-7f5b-479f-b795-de095602ac80" (UID: "af08d05c-7f5b-479f-b795-de095602ac80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:41.452237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.452204 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:41.454317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.454290 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv" pod "af08d05c-7f5b-479f-b795-de095602ac80" (UID: "af08d05c-7f5b-479f-b795-de095602ac80"). InnerVolumeSpecName "kube-api-access-jsbkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:41.454668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.454642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca" pod "af08d05c-7f5b-479f-b795-de095602ac80" (UID: "af08d05c-7f5b-479f-b795-de095602ac80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:41.553289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.553264 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbkv\" (UniqueName: \"kubernetes.io/projected/af08d05c-7f5b-479f-b795-de095602ac80-kube-api-access-jsbkv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:41.553289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:41.553289 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af08d05c-7f5b-479f-b795-de095602ac80-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:42.171508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.170812 4543 generic.go:356] "Generic (PLEG): container finished" podID="c6109265-a586-4622-90eb-2a75272e6a37" containerID="23548bf032dca6308f3a56d74af10a79a4331cd2ddfe19d7bdd1f8b3753e696d" exitCode=0 Oct 29 22:09:42.171508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.170926 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" event={"ID":"c6109265-a586-4622-90eb-2a75272e6a37","Type":"ContainerDied","Data":"23548bf032dca6308f3a56d74af10a79a4331cd2ddfe19d7bdd1f8b3753e696d"} Oct 29 22:09:42.173976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.173950 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" Oct 29 22:09:42.174168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.174099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp" event={"ID":"af08d05c-7f5b-479f-b795-de095602ac80","Type":"ContainerDied","Data":"bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903"} Oct 29 22:09:42.174168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.174141 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc786cf24c9c733492126880c55ad8a05e5404f122380bacb6ad8f4889b68903" Oct 29 22:09:42.662305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.662271 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn"] Oct 29 22:09:42.667494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:42.667467 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6153f8-z6fwn"] Oct 29 22:09:43.341857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.341826 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:43.379180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.379156 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\" (UniqueName: \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\") pod \"c6109265-a586-4622-90eb-2a75272e6a37\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " Oct 29 22:09:43.379289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.379203 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\") pod \"c6109265-a586-4622-90eb-2a75272e6a37\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " Oct 29 22:09:43.379352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.379320 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\") pod \"c6109265-a586-4622-90eb-2a75272e6a37\" (UID: \"c6109265-a586-4622-90eb-2a75272e6a37\") " Oct 29 22:09:43.379534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.379511 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca" pod "c6109265-a586-4622-90eb-2a75272e6a37" (UID: "c6109265-a586-4622-90eb-2a75272e6a37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:43.379811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.379793 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:43.381393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.381372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca" pod "c6109265-a586-4622-90eb-2a75272e6a37" (UID: "c6109265-a586-4622-90eb-2a75272e6a37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:43.381455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.381406 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj" pod "c6109265-a586-4622-90eb-2a75272e6a37" (UID: "c6109265-a586-4622-90eb-2a75272e6a37"). InnerVolumeSpecName "kube-api-access-5cvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:43.481741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.481717 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvrj\" (UniqueName: \"kubernetes.io/projected/c6109265-a586-4622-90eb-2a75272e6a37-kube-api-access-5cvrj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:43.481856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:43.481755 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6109265-a586-4622-90eb-2a75272e6a37-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:44.191414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:44.191385 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" Oct 29 22:09:44.191414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:44.191406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52" event={"ID":"c6109265-a586-4622-90eb-2a75272e6a37","Type":"ContainerDied","Data":"9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175"} Oct 29 22:09:44.191680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:44.191456 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b96ab5bedeefb6beb10b4eb75e14427d0524fadbc229691db6a7a4052e7a175" Oct 29 22:09:44.386925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:44.386882 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c79b9f-2563-46d1-ad53-c9a07337826a" path="/var/lib/kubelet/pods/d2c79b9f-2563-46d1-ad53-c9a07337826a/volumes" Oct 29 22:09:48.633821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.633786 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697"] Oct 29 22:09:48.634496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634470 4543 cpu_manager.go:414] "containerMap: removing container" podUID="af08d05c-7f5b-479f-b795-de095602ac80" containerName="revoke-keys" Oct 29 22:09:48.634496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634490 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af08d05c-7f5b-479f-b795-de095602ac80" containerName="revoke-keys" Oct 29 22:09:48.634712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634536 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c6109265-a586-4622-90eb-2a75272e6a37" containerName="revoke-keys" Oct 29 22:09:48.634712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634545 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6109265-a586-4622-90eb-2a75272e6a37" containerName="revoke-keys" Oct 29 22:09:48.634819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634737 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6109265-a586-4622-90eb-2a75272e6a37" containerName="revoke-keys" Oct 29 22:09:48.634819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.634761 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="af08d05c-7f5b-479f-b795-de095602ac80" containerName="revoke-keys" Oct 29 22:09:48.639227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.639205 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.657532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.657503 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697"] Oct 29 22:09:48.688813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.688789 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:09:48.689009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.688789 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:09:48.689124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.688827 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:09:48.689124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.688904 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:09:48.790819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.790781 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.791109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.791070 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zhh\" (UniqueName: \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.791194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.791169 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.892838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.892753 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zhh\" (UniqueName: \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.892838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.892802 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.893026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.892901 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.893576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.893553 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.895363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.895336 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.919593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.919572 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zhh\" (UniqueName: \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:48.955508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:48.955491 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:49.267477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:49.267446 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697"] Oct 29 22:09:49.272379 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:49.272350 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2c4081_56b2_4824_9ca9_9fa3261c63e6.slice/crio-05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094 WatchSource:0}: Error finding container 05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094: Status 404 returned error can't find the container with id 05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094 Oct 29 22:09:50.237229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:50.237196 4543 generic.go:356] "Generic (PLEG): container finished" podID="8f2c4081-56b2-4824-9ca9-9fa3261c63e6" containerID="4b5487a28253ebcdf76ee636cfaf8a5df35dacd0d30bacc776c09b3bb706af5d" exitCode=0 Oct 29 22:09:50.237527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:50.237273 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" event={"ID":"8f2c4081-56b2-4824-9ca9-9fa3261c63e6","Type":"ContainerDied","Data":"4b5487a28253ebcdf76ee636cfaf8a5df35dacd0d30bacc776c09b3bb706af5d"} Oct 29 22:09:50.237527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:50.237317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" event={"ID":"8f2c4081-56b2-4824-9ca9-9fa3261c63e6","Type":"ContainerStarted","Data":"05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094"} Oct 29 22:09:51.332351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.332320 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:09:51.332764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.332570 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" containerID="cri-o://0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0" gracePeriod=30 Oct 29 22:09:51.553575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.553184 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:51.618864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.618831 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5"] Oct 29 22:09:51.619793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.619772 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8f2c4081-56b2-4824-9ca9-9fa3261c63e6" containerName="revoke-keys" Oct 29 22:09:51.619923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.619909 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2c4081-56b2-4824-9ca9-9fa3261c63e6" containerName="revoke-keys" Oct 29 22:09:51.620207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.620195 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2c4081-56b2-4824-9ca9-9fa3261c63e6" containerName="revoke-keys" Oct 29 22:09:51.626587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.626567 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.645556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.645518 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5"] Oct 29 22:09:51.686502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.686469 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\") pod \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " Oct 29 22:09:51.686675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.686657 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\" (UniqueName: \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\") pod \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " Oct 29 22:09:51.686831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.686817 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\") pod \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\" (UID: \"8f2c4081-56b2-4824-9ca9-9fa3261c63e6\") " Oct 29 22:09:51.686918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.686879 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca" pod "8f2c4081-56b2-4824-9ca9-9fa3261c63e6" (UID: "8f2c4081-56b2-4824-9ca9-9fa3261c63e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:51.687068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.687049 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7x7l\" (UniqueName: \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.687230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.687195 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.687342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.687279 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.687472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.687451 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:51.689554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.689517 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh" pod "8f2c4081-56b2-4824-9ca9-9fa3261c63e6" (UID: "8f2c4081-56b2-4824-9ca9-9fa3261c63e6"). InnerVolumeSpecName "kube-api-access-j4zhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:51.689554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.689520 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca" pod "8f2c4081-56b2-4824-9ca9-9fa3261c63e6" (UID: "8f2c4081-56b2-4824-9ca9-9fa3261c63e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:51.788534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.788506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.788666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.788654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7x7l\" (UniqueName: \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.788711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.788704 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.788761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.788752 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:51.788798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.788764 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zhh\" (UniqueName: \"kubernetes.io/projected/8f2c4081-56b2-4824-9ca9-9fa3261c63e6-kube-api-access-j4zhh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:51.789163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.789143 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.791034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.791016 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.814662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.814616 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7x7l\" (UniqueName: \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\") pod \"maas-api-revoke-sub-ecdb9c5c-6kbn5\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:51.949255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:51.949222 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:52.254518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:52.254485 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" Oct 29 22:09:52.254518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:52.254507 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697" event={"ID":"8f2c4081-56b2-4824-9ca9-9fa3261c63e6","Type":"ContainerDied","Data":"05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094"} Oct 29 22:09:52.254732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:52.254555 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05cfa53288080c038cdc01470af88f58265510f89d7263c89654cf0f31744094" Oct 29 22:09:52.264091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:52.264050 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5"] Oct 29 22:09:52.264226 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:52.264206 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28110085_6c20_44d7_b72a_3d3b28a76440.slice/crio-f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb WatchSource:0}: Error finding container f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb: Status 404 returned error can't find the container with id f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb Oct 29 22:09:53.265272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.265234 4543 generic.go:356] "Generic (PLEG): container finished" podID="28110085-6c20-44d7-b72a-3d3b28a76440" containerID="25babdf5505e6325ee657034bcaeff13e6f032e20e7b140e6b648014f689aecc" exitCode=0 Oct 29 22:09:53.265653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.265314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" event={"ID":"28110085-6c20-44d7-b72a-3d3b28a76440","Type":"ContainerDied","Data":"25babdf5505e6325ee657034bcaeff13e6f032e20e7b140e6b648014f689aecc"} Oct 29 22:09:53.265653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.265373 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" event={"ID":"28110085-6c20-44d7-b72a-3d3b28a76440","Type":"ContainerStarted","Data":"f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb"} Oct 29 22:09:53.396176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.396093 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:09:53.400659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.400634 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.418291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.418266 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-80bd36-openshift-default-dockercfg-lkxsj\"" Oct 29 22:09:53.425153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.425131 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:09:53.528236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.528132 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.528556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.528533 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.528740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.528723 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.528968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.528950 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.529149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.529133 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.543096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.529320 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.543510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.543486 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.543760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.543741 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.543937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.543920 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvp6\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654581 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654667 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654716 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvp6\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654820 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654905 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654972 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.654996 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.655051 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.655900 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.658146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.656924 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.659506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.659481 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.663543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.659930 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.663543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.660916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.664315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.664266 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.664553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.664531 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.685543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.685516 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvp6\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.686157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.686139 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\") pod \"e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.727036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.727014 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:53.922121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:53.922018 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx_0ed3194e-7db5-40f4-8b9e-d925e7f051ab/payload-processing/0.log" Oct 29 22:09:54.075186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.075155 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:09:54.075970 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:54.075937 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e455388_f6b8_4b18_b526_cc3807ad7c79.slice/crio-bf02eb19aa7d4736d29fa27dbd6e8f01faaae5b18076b2abfa308d66d01aecd0 WatchSource:0}: Error finding container bf02eb19aa7d4736d29fa27dbd6e8f01faaae5b18076b2abfa308d66d01aecd0: Status 404 returned error can't find the container with id bf02eb19aa7d4736d29fa27dbd6e8f01faaae5b18076b2abfa308d66d01aecd0 Oct 29 22:09:54.078753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.078716 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:54.078861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.078784 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:54.078861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.078814 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:09:54.126212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.126170 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx_0ed3194e-7db5-40f4-8b9e-d925e7f051ab/payload-processing/0.log" Oct 29 22:09:54.274660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.274609 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerStarted","Data":"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83"} Oct 29 22:09:54.275059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.274666 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerStarted","Data":"bf02eb19aa7d4736d29fa27dbd6e8f01faaae5b18076b2abfa308d66d01aecd0"} Oct 29 22:09:54.326487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.326425 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podStartSLOduration=1.3264054220000001 podStartE2EDuration="1.326405422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:54.322865036 +0000 UTC m=+1656.064795709" watchObservedRunningTime="2026-10-29 22:09:54.326405422 +0000 UTC m=+1656.068336099" Oct 29 22:09:54.419902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.419877 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:54.571667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.571579 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\") pod \"28110085-6c20-44d7-b72a-3d3b28a76440\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " Oct 29 22:09:54.571667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.571656 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\") pod \"28110085-6c20-44d7-b72a-3d3b28a76440\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " Oct 29 22:09:54.571980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.571697 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\" (UniqueName: \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\") pod \"28110085-6c20-44d7-b72a-3d3b28a76440\" (UID: \"28110085-6c20-44d7-b72a-3d3b28a76440\") " Oct 29 22:09:54.572133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.572107 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca" pod "28110085-6c20-44d7-b72a-3d3b28a76440" (UID: "28110085-6c20-44d7-b72a-3d3b28a76440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:54.572439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.572417 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.573978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.573945 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l" pod "28110085-6c20-44d7-b72a-3d3b28a76440" (UID: "28110085-6c20-44d7-b72a-3d3b28a76440"). InnerVolumeSpecName "kube-api-access-g7x7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:54.574144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.574114 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca" pod "28110085-6c20-44d7-b72a-3d3b28a76440" (UID: "28110085-6c20-44d7-b72a-3d3b28a76440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:54.673530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.673504 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7x7l\" (UniqueName: \"kubernetes.io/projected/28110085-6c20-44d7-b72a-3d3b28a76440-kube-api-access-g7x7l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.673637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.673536 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28110085-6c20-44d7-b72a-3d3b28a76440-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:54.727766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.727737 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:54.917653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.917581 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng"] Oct 29 22:09:54.918290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.918275 4543 cpu_manager.go:414] "containerMap: removing container" podUID="28110085-6c20-44d7-b72a-3d3b28a76440" containerName="revoke-keys" Oct 29 22:09:54.918343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.918292 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28110085-6c20-44d7-b72a-3d3b28a76440" containerName="revoke-keys" Oct 29 22:09:54.918441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.918432 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="28110085-6c20-44d7-b72a-3d3b28a76440" containerName="revoke-keys" Oct 29 22:09:54.922236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.922218 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:54.939643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.939619 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng"] Oct 29 22:09:54.977717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.977690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qr2\" (UniqueName: \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:54.977975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.977948 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:54.978189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:54.978166 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.080156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.080118 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.080258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.080199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.080461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.080444 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qr2\" (UniqueName: \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.080804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.080785 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.082623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.082604 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.106205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.106185 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qr2\" (UniqueName: \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.237436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.237404 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:55.283927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.283895 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" Oct 29 22:09:55.284422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.283931 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5" event={"ID":"28110085-6c20-44d7-b72a-3d3b28a76440","Type":"ContainerDied","Data":"f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb"} Oct 29 22:09:55.284422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.283976 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91b8ad3462abdeac462fb3f70961375619134084f6396bf9fbfdd8c958babdb" Oct 29 22:09:55.552339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.552317 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng"] Oct 29 22:09:55.553121 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:55.553062 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459137e4_1986_441c_b87a_c093dbbd74e0.slice/crio-bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a WatchSource:0}: Error finding container bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a: Status 404 returned error can't find the container with id bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a Oct 29 22:09:55.734244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:55.734220 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:56.293669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.293637 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" event={"ID":"459137e4-1986-441c-b87a-c093dbbd74e0","Type":"ContainerStarted","Data":"8acd857698394938a1cbf89f39c8431db00dc2f60ce6d53909983fb891768a74"} Oct 29 22:09:56.294169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.293695 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" event={"ID":"459137e4-1986-441c-b87a-c093dbbd74e0","Type":"ContainerStarted","Data":"bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a"} Oct 29 22:09:56.294252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.294163 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:56.295022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.295004 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:09:56.338592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.338550 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" podStartSLOduration=2.338537475 podStartE2EDuration="2.338537475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:56.337314368 +0000 UTC m=+1658.079245041" watchObservedRunningTime="2026-10-29 22:09:56.338537475 +0000 UTC m=+1658.080468186" Oct 29 22:09:56.619069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.619040 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:09:56.712599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712573 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.712749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712646 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.712749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712668 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.712895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.712895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712795 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.712895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712854 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.713063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712894 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.713063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712968 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.713063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.712997 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:56.713249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713138 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\") pod \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\" (UID: \"9c1123aa-5079-4e61-bfdc-e670b00d78c9\") " Oct 29 22:09:56.713406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713380 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:56.713754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713719 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:56.713754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713745 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.713917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713767 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.713917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713869 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:56.714034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.713929 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:56.715746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.715715 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:09:56.715839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.715759 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:09:56.716935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.716910 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:56.717048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.716948 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd" pod "9c1123aa-5079-4e61-bfdc-e670b00d78c9" (UID: "9c1123aa-5079-4e61-bfdc-e670b00d78c9"). InnerVolumeSpecName "kube-api-access-nt6fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:56.815432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815407 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815434 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815447 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815461 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815473 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815484 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1123aa-5079-4e61-bfdc-e670b00d78c9-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:56.815594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:56.815496 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6fd\" (UniqueName: \"kubernetes.io/projected/9c1123aa-5079-4e61-bfdc-e670b00d78c9-kube-api-access-nt6fd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:57.319717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.319673 4543 generic.go:356] "Generic (PLEG): container finished" podID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerID="0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0" exitCode=0 Oct 29 22:09:57.320415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.320386 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerDied","Data":"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0"} Oct 29 22:09:57.320712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.320686 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" event={"ID":"9c1123aa-5079-4e61-bfdc-e670b00d78c9","Type":"ContainerDied","Data":"493f6ff7d363d4a9b89268454860b9becac039036f1ddd2129a8b23b854a2325"} Oct 29 22:09:57.320962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.320936 4543 scope.go:122] "RemoveContainer" containerID="0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0" Oct 29 22:09:57.323431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.322449 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws" Oct 29 22:09:57.332432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.332389 4543 generic.go:356] "Generic (PLEG): container finished" podID="459137e4-1986-441c-b87a-c093dbbd74e0" containerID="8acd857698394938a1cbf89f39c8431db00dc2f60ce6d53909983fb891768a74" exitCode=0 Oct 29 22:09:57.332711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.332669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" event={"ID":"459137e4-1986-441c-b87a-c093dbbd74e0","Type":"ContainerDied","Data":"8acd857698394938a1cbf89f39c8431db00dc2f60ce6d53909983fb891768a74"} Oct 29 22:09:57.347446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.347381 4543 scope.go:122] "RemoveContainer" containerID="a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b" Oct 29 22:09:57.360894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.360874 4543 scope.go:122] "RemoveContainer" containerID="0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0" Oct 29 22:09:57.361538 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:57.361511 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0\": container with ID starting with 0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0 not found: ID does not exist" containerID="0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0" Oct 29 22:09:57.361661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.361546 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0"} err="failed to get container status \"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0\": rpc error: code = NotFound desc = could not find container \"0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0\": container with ID starting with 0b3c3a602c8f40fafbd2d4f15aaa7d0393f02faf1d8f15db54f3f1a0cddbeee0 not found: ID does not exist" Oct 29 22:09:57.361661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.361566 4543 scope.go:122] "RemoveContainer" containerID="a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b" Oct 29 22:09:57.361873 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:57.361855 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b\": container with ID starting with a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b not found: ID does not exist" containerID="a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b" Oct 29 22:09:57.361941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.361876 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b"} err="failed to get container status \"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b\": rpc error: code = NotFound desc = could not find container \"a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b\": container with ID starting with a2fdf30c9cc9428bbc2ea07a1d6fe24d971831dc6735245af45adeb4111e756b not found: ID does not exist" Oct 29 22:09:57.429844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.429779 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:09:57.443194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:57.443166 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6153f8-openshift-default-79c746bcf7-69vws"] Oct 29 22:09:57.961794 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:09:57.961753 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:09:58.387326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.387295 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" path="/var/lib/kubelet/pods/9c1123aa-5079-4e61-bfdc-e670b00d78c9/volumes" Oct 29 22:09:58.509205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.509175 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:58.642150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.642042 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\") pod \"459137e4-1986-441c-b87a-c093dbbd74e0\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " Oct 29 22:09:58.642315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.642147 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\" (UniqueName: \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\") pod \"459137e4-1986-441c-b87a-c093dbbd74e0\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " Oct 29 22:09:58.642315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.642257 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\") pod \"459137e4-1986-441c-b87a-c093dbbd74e0\" (UID: \"459137e4-1986-441c-b87a-c093dbbd74e0\") " Oct 29 22:09:58.642608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.642581 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca" pod "459137e4-1986-441c-b87a-c093dbbd74e0" (UID: "459137e4-1986-441c-b87a-c093dbbd74e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:58.642913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.642760 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:58.644300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.644279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca" pod "459137e4-1986-441c-b87a-c093dbbd74e0" (UID: "459137e4-1986-441c-b87a-c093dbbd74e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:58.644551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.644525 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2" pod "459137e4-1986-441c-b87a-c093dbbd74e0" (UID: "459137e4-1986-441c-b87a-c093dbbd74e0"). InnerVolumeSpecName "kube-api-access-l7qr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:58.692939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.692786 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694066 4543 cpu_manager.go:414] "containerMap: removing container" podUID="459137e4-1986-441c-b87a-c093dbbd74e0" containerName="revoke-keys" Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694119 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459137e4-1986-441c-b87a-c093dbbd74e0" containerName="revoke-keys" Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694157 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694167 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694357 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.711256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.694377 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="459137e4-1986-441c-b87a-c093dbbd74e0" containerName="revoke-keys" Oct 29 22:09:58.854233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.820730 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459137e4-1986-441c-b87a-c093dbbd74e0-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:58.854233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.820768 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qr2\" (UniqueName: \"kubernetes.io/projected/459137e4-1986-441c-b87a-c093dbbd74e0-kube-api-access-l7qr2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:09:58.876429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.873545 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:09:58.876429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.873666 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:58.882684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.882477 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:09:58.885837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.883549 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.885837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.883574 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.885837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.883804 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1123aa-5079-4e61-bfdc-e670b00d78c9" containerName="istio-proxy" Oct 29 22:09:58.897276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.897253 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-80bd36\"" Oct 29 22:09:58.900069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.900050 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:09:58.900223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.900173 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:58.917573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.917548 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-80bd36-dockercfg-t5n7j\"" Oct 29 22:09:58.922273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.922246 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:58.922393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:58.922379 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvs9\" (UniqueName: \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.023326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.023289 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.023504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.023392 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.023555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.023510 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvs9\" (UniqueName: \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.023614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.023549 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfvm\" (UniqueName: \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.024321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.024301 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.050113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.050062 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvs9\" (UniqueName: \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\") pod \"payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.124827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.124791 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.124937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.124906 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfvm\" (UniqueName: \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.125517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.125494 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.150690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.150633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfvm\" (UniqueName: \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\") pod \"payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.202839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.202816 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:09:59.215434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.215419 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:09:59.352515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.352481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" event={"ID":"459137e4-1986-441c-b87a-c093dbbd74e0","Type":"ContainerDied","Data":"bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a"} Oct 29 22:09:59.352693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.352533 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1364a5aba11f389cef9b17661740b951665ef93f0277cb0d87cf458d85809a" Oct 29 22:09:59.352693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.352584 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng" Oct 29 22:09:59.523121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.523093 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:09:59.523916 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:59.523889 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33be8328_cdbb_4f7f_8d6e_4ec08bce7af7.slice/crio-ea0dce03ace66a6fee858abe72781f567b03f30c6fa5b663ea707149010b5963 WatchSource:0}: Error finding container ea0dce03ace66a6fee858abe72781f567b03f30c6fa5b663ea707149010b5963: Status 404 returned error can't find the container with id ea0dce03ace66a6fee858abe72781f567b03f30c6fa5b663ea707149010b5963 Oct 29 22:09:59.536966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.536945 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:09:59.539631 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:09:59.539604 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98860da_1de2_489d_aeaf_e1968915a001.slice/crio-d39207c6a5f1891e80df7ada63d81b939453f283d251345994a7f4e3e3215076 WatchSource:0}: Error finding container d39207c6a5f1891e80df7ada63d81b939453f283d251345994a7f4e3e3215076: Status 404 returned error can't find the container with id d39207c6a5f1891e80df7ada63d81b939453f283d251345994a7f4e3e3215076 Oct 29 22:09:59.743924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.743894 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547"] Oct 29 22:09:59.748230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.748213 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.766268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.766246 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:09:59.766372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.766290 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:09:59.766579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.766564 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:09:59.766628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.766618 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:09:59.780744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.780692 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547"] Oct 29 22:09:59.835427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.835404 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.835629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.835609 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlv2\" (UniqueName: \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.835759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.835745 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.936496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.936469 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlv2\" (UniqueName: \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.936612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.936585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.936680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.936617 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.937324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.937298 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.939365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.939346 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:09:59.962302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:09:59.962279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlv2\" (UniqueName: \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\") pod \"maas-api-revoke-sub-7257ec0f-wr547\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:10:00.062215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.062164 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:10:00.361676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.361580 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" event={"ID":"b98860da-1de2-489d-aeaf-e1968915a001","Type":"ContainerStarted","Data":"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117"} Oct 29 22:10:00.361829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.361684 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:10:00.361829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.361708 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" event={"ID":"b98860da-1de2-489d-aeaf-e1968915a001","Type":"ContainerStarted","Data":"d39207c6a5f1891e80df7ada63d81b939453f283d251345994a7f4e3e3215076"} Oct 29 22:10:00.363340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.363315 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" event={"ID":"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7","Type":"ContainerStarted","Data":"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190"} Oct 29 22:10:00.363452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.363354 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" event={"ID":"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7","Type":"ContainerStarted","Data":"ea0dce03ace66a6fee858abe72781f567b03f30c6fa5b663ea707149010b5963"} Oct 29 22:10:00.363452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.363436 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:10:00.369953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.369930 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547"] Oct 29 22:10:00.370613 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:00.370588 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fed8d4_2429_48a0_bbf6_e70608e178d7.slice/crio-5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe WatchSource:0}: Error finding container 5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe: Status 404 returned error can't find the container with id 5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe Oct 29 22:10:00.408592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.408551 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" podStartSLOduration=2.408539997 podStartE2EDuration="2.408539997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:00.404675041 +0000 UTC m=+1662.146605714" watchObservedRunningTime="2026-10-29 22:10:00.408539997 +0000 UTC m=+1662.150470709" Oct 29 22:10:00.452174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:00.452131 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" podStartSLOduration=2.452119675 podStartE2EDuration="2.452119675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:00.447639126 +0000 UTC m=+1662.189569790" watchObservedRunningTime="2026-10-29 22:10:00.452119675 +0000 UTC m=+1662.194050348" Oct 29 22:10:01.373446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.373414 4543 generic.go:356] "Generic (PLEG): container finished" podID="25fed8d4-2429-48a0-bbf6-e70608e178d7" containerID="b78f969dccd54dbb23efebd5a53c5bf11c0e007259b6614693adf21c88191360" exitCode=0 Oct 29 22:10:01.373843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.373513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" event={"ID":"25fed8d4-2429-48a0-bbf6-e70608e178d7","Type":"ContainerDied","Data":"b78f969dccd54dbb23efebd5a53c5bf11c0e007259b6614693adf21c88191360"} Oct 29 22:10:01.373843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.373577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" event={"ID":"25fed8d4-2429-48a0-bbf6-e70608e178d7","Type":"ContainerStarted","Data":"5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe"} Oct 29 22:10:01.963191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.963162 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz"] Oct 29 22:10:01.967774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.967758 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:01.990725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:01.990698 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz"] Oct 29 22:10:02.059889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.059863 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.060033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.059895 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.060033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.059980 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9bn\" (UniqueName: \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.161205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.161177 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.161393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.161214 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.161393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.161278 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9bn\" (UniqueName: \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.161914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.161890 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.163602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.163582 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.187161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.187134 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9bn\" (UniqueName: \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\") pod \"maas-api-revoke-sub-05283aed-dddnz\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.282129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.282036 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:02.529660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.529637 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:10:02.593484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.593458 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz"] Oct 29 22:10:02.594680 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:02.594655 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b831047_e15d_424b_9703_58b650108c17.slice/crio-fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2 WatchSource:0}: Error finding container fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2: Status 404 returned error can't find the container with id fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2 Oct 29 22:10:02.670207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.670183 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\") pod \"25fed8d4-2429-48a0-bbf6-e70608e178d7\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " Oct 29 22:10:02.670303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.670238 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\" (UniqueName: \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\") pod \"25fed8d4-2429-48a0-bbf6-e70608e178d7\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " Oct 29 22:10:02.670468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.670454 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\") pod \"25fed8d4-2429-48a0-bbf6-e70608e178d7\" (UID: \"25fed8d4-2429-48a0-bbf6-e70608e178d7\") " Oct 29 22:10:02.670515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.670480 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca" pod "25fed8d4-2429-48a0-bbf6-e70608e178d7" (UID: "25fed8d4-2429-48a0-bbf6-e70608e178d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:02.670990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.670973 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:02.672430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.672405 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2" pod "25fed8d4-2429-48a0-bbf6-e70608e178d7" (UID: "25fed8d4-2429-48a0-bbf6-e70608e178d7"). InnerVolumeSpecName "kube-api-access-zmlv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:02.672547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.672529 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca" pod "25fed8d4-2429-48a0-bbf6-e70608e178d7" (UID: "25fed8d4-2429-48a0-bbf6-e70608e178d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:02.772646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.772624 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fed8d4-2429-48a0-bbf6-e70608e178d7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:02.772646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:02.772647 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmlv2\" (UniqueName: \"kubernetes.io/projected/25fed8d4-2429-48a0-bbf6-e70608e178d7-kube-api-access-zmlv2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:03.391917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.391890 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" Oct 29 22:10:03.392059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.391890 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547" event={"ID":"25fed8d4-2429-48a0-bbf6-e70608e178d7","Type":"ContainerDied","Data":"5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe"} Oct 29 22:10:03.392059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.392012 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe5cfa68c6f6e9355c06425a73e6ca40a49eb7b26ab542fc44ca815b43ac7fe" Oct 29 22:10:03.393854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.393826 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" event={"ID":"1b831047-e15d-424b-9703-58b650108c17","Type":"ContainerStarted","Data":"cee23880f3fdcdae9c4b9d9e0f9c3524d71a5ab18521bdba1c579b788eee75fb"} Oct 29 22:10:03.393967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.393880 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" event={"ID":"1b831047-e15d-424b-9703-58b650108c17","Type":"ContainerStarted","Data":"fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2"} Oct 29 22:10:03.438654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:03.438586 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" podStartSLOduration=2.438575717 podStartE2EDuration="2.438575717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:03.438181137 +0000 UTC m=+1665.180111811" watchObservedRunningTime="2026-10-29 22:10:03.438575717 +0000 UTC m=+1665.180506390" Oct 29 22:10:04.403542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:04.403504 4543 generic.go:356] "Generic (PLEG): container finished" podID="1b831047-e15d-424b-9703-58b650108c17" containerID="cee23880f3fdcdae9c4b9d9e0f9c3524d71a5ab18521bdba1c579b788eee75fb" exitCode=0 Oct 29 22:10:04.403910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:04.403593 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" event={"ID":"1b831047-e15d-424b-9703-58b650108c17","Type":"ContainerDied","Data":"cee23880f3fdcdae9c4b9d9e0f9c3524d71a5ab18521bdba1c579b788eee75fb"} Oct 29 22:10:05.589055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.589030 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:05.708885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.708825 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\" (UniqueName: \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\") pod \"1b831047-e15d-424b-9703-58b650108c17\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " Oct 29 22:10:05.709033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.708947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\") pod \"1b831047-e15d-424b-9703-58b650108c17\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " Oct 29 22:10:05.709118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.709094 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\") pod \"1b831047-e15d-424b-9703-58b650108c17\" (UID: \"1b831047-e15d-424b-9703-58b650108c17\") " Oct 29 22:10:05.709934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.709460 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca" pod "1b831047-e15d-424b-9703-58b650108c17" (UID: "1b831047-e15d-424b-9703-58b650108c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:05.709934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.709899 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b831047-e15d-424b-9703-58b650108c17-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:05.711480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.711434 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn" pod "1b831047-e15d-424b-9703-58b650108c17" (UID: "1b831047-e15d-424b-9703-58b650108c17"). InnerVolumeSpecName "kube-api-access-6k9bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:05.711753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.711720 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca" pod "1b831047-e15d-424b-9703-58b650108c17" (UID: "1b831047-e15d-424b-9703-58b650108c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:05.810788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.810764 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k9bn\" (UniqueName: \"kubernetes.io/projected/1b831047-e15d-424b-9703-58b650108c17-kube-api-access-6k9bn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:05.810788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:05.810787 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b831047-e15d-424b-9703-58b650108c17-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:06.420496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:06.420466 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" Oct 29 22:10:06.420626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:06.420498 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz" event={"ID":"1b831047-e15d-424b-9703-58b650108c17","Type":"ContainerDied","Data":"fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2"} Oct 29 22:10:06.420626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:06.420550 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcc494374a4137d9862d6f53dafab3817e777167339e6281e541dfaaf1eca1f2" Oct 29 22:10:07.982645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.982613 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:10:07.983575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983542 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1b831047-e15d-424b-9703-58b650108c17" containerName="revoke-keys" Oct 29 22:10:07.983575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983569 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b831047-e15d-424b-9703-58b650108c17" containerName="revoke-keys" Oct 29 22:10:07.983751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983637 4543 cpu_manager.go:414] "containerMap: removing container" podUID="25fed8d4-2429-48a0-bbf6-e70608e178d7" containerName="revoke-keys" Oct 29 22:10:07.983751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983646 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25fed8d4-2429-48a0-bbf6-e70608e178d7" containerName="revoke-keys" Oct 29 22:10:07.983819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983776 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="25fed8d4-2429-48a0-bbf6-e70608e178d7" containerName="revoke-keys" Oct 29 22:10:07.983819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.983790 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b831047-e15d-424b-9703-58b650108c17" containerName="revoke-keys" Oct 29 22:10:07.987617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:07.987598 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.006189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.006169 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fe45b-openshift-default-dockercfg-kkxxf\"" Oct 29 22:10:08.024335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.024305 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:10:08.141754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141723 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.141882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141779 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.141882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141869 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.141997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141926 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vmz\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.141997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141952 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.141997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.141991 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.142165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.142143 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.142235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.142224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.142269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.142258 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243605 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243690 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243719 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243803 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243860 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243917 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.243984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.243978 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vmz\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244017 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244049 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244098 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244107 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244209 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.244678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.244654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.246241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.246221 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.246547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.246526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.269558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.269535 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.269799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.269780 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vmz\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\") pod \"e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.303174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.303152 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:08.630768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.630742 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:10:08.630880 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:08.630816 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105252e5_1578_4832_91c0_0a1e8303b177.slice/crio-273abcca580824db210c8421d182b7e1895e96b2f63729c8cc3bf48f3ce83d40 WatchSource:0}: Error finding container 273abcca580824db210c8421d182b7e1895e96b2f63729c8cc3bf48f3ce83d40: Status 404 returned error can't find the container with id 273abcca580824db210c8421d182b7e1895e96b2f63729c8cc3bf48f3ce83d40 Oct 29 22:10:08.633362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.633328 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:08.633466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.633400 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:08.633466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:08.633445 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:09.461769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:09.461716 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" event={"ID":"105252e5-1578-4832-91c0-0a1e8303b177","Type":"ContainerStarted","Data":"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b"} Oct 29 22:10:09.462215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:09.461792 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" event={"ID":"105252e5-1578-4832-91c0-0a1e8303b177","Type":"ContainerStarted","Data":"273abcca580824db210c8421d182b7e1895e96b2f63729c8cc3bf48f3ce83d40"} Oct 29 22:10:09.555971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:09.555901 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podStartSLOduration=2.555879497 podStartE2EDuration="2.555879497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:09.523663737 +0000 UTC m=+1671.265594410" watchObservedRunningTime="2026-10-29 22:10:09.555879497 +0000 UTC m=+1671.297810172" Oct 29 22:10:10.304141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:10.304105 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:10.305630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:10.305602 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:10.305769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:10.305665 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:11.593099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:11.464770 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:11.593099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:11.464840 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:11.593099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:11.563591 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:10:11.610102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:11.604584 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:10:12.074385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.072204 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:10:12.132102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.111334 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.132102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.120210 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.193043 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.206101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.194789 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:10:12.259243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.183307 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.363528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.363438 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:12.363760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.363731 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:12.482227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.482177 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.555119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.555058 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.631159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.616773 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.631159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.620565 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697"] Oct 29 22:10:12.662174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:12.650869 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c53cc8-5p697"] Oct 29 22:10:13.074102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.072596 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:10:13.082099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.078461 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:10:13.119348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.116937 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.119348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.118162 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:10:13.122844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.122814 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.123836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.123815 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:10:13.324164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.280151 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:10:13.343445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.282022 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:10:13.370099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.345494 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.535997 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.536097 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.536147 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.536175 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.536207 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:10:13.536403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.525414 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.536878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.330187 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:13.536878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.536552 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:13.549166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.282055 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:10:13.549166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.367761 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Oct 29 22:10:13.549166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.435386 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" podUID="543853b5-9761-4852-8801-1519786ced31" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.549166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.539016 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:10:13.603226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.539066 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:10:13.663098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.629393 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.663098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.629751 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:10:13.663098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.588589 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:10:13.663098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.636431 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:10:13.680872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.680846 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:10:13.721221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:13.721176 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4rnjf" podUID="543853b5-9761-4852-8801-1519786ced31" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.118104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.117586 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:10:14.313757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.312575 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:14.313757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.312628 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:14.497406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.497372 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2c4081-56b2-4824-9ca9-9fa3261c63e6" path="/var/lib/kubelet/pods/8f2c4081-56b2-4824-9ca9-9fa3261c63e6/volumes" Oct 29 22:10:14.510758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.510731 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:10:14.568233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.568184 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.568800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.568770 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:10:14.724338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.724182 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:10:14.796322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.796174 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:10:14.796322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.796244 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.802517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:14.802485 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:10:15.303798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:15.303759 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:15.304524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:15.304487 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:16.304282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:16.304245 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 22:10:16.304668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:16.304302 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 22:10:17.308326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:17.308293 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:17.308795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:17.308776 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:17.309380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:17.309357 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:10:18.202978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.202946 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 22:10:18.203182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.203152 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" containerID="cri-o://576001ca5abdc1550ddbeba7caf3dff34d24163e7c0054a9752cb007a6a44ef1" gracePeriod=30 Oct 29 22:10:18.222695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.222665 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:10:18.228720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.228700 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.239788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.239759 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:10:18.244126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.244102 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.252448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.252421 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:10:18.283862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.283837 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:10:18.296506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296463 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.296636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296519 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.296698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296640 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.296698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zjw\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.296886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.296958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296883 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.296958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296939 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.297107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.296978 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.297107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297001 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297070 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzvv\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297123 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297182 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.297285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297223 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297253 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.297509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297323 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.297509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297371 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.297509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.297408 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398353 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398407 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398463 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzvv\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398485 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398582 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398624 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398679 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398706 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398773 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.398823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398826 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.398836 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399152 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399190 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399211 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399218 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zjw\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399487 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399588 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.399853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399620 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.399947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.399926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.400180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.400155 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.400333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.400310 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.400431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.400179 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.400600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.400576 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.401354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.401332 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.401587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.401568 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.401830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.401814 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.402712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.402692 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.423709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.423688 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzvv\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.424146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.424124 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.424363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.424344 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.424924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.424907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zjw\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.561047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.561024 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:18.576993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.576969 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:18.895134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.895104 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:10:18.895366 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:18.895343 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699c48b1_116c_4902_9025_0505203814fd.slice/crio-db42a11bfcaecc3823292dd2defb4679661cd76ae5cfc1e86c0b0c5dba6acc74 WatchSource:0}: Error finding container db42a11bfcaecc3823292dd2defb4679661cd76ae5cfc1e86c0b0c5dba6acc74: Status 404 returned error can't find the container with id db42a11bfcaecc3823292dd2defb4679661cd76ae5cfc1e86c0b0c5dba6acc74 Oct 29 22:10:18.897880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.897848 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:18.897952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.897914 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:18.897952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.897944 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:18.912122 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:18.912089 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957524f0_a0ac_49fa_9dfd_50ff2fa43a52.slice/crio-3355540f6b57c58b336307c4b0dd4e669855bc1d971c917278a8beb3f7175d29 WatchSource:0}: Error finding container 3355540f6b57c58b336307c4b0dd4e669855bc1d971c917278a8beb3f7175d29: Status 404 returned error can't find the container with id 3355540f6b57c58b336307c4b0dd4e669855bc1d971c917278a8beb3f7175d29 Oct 29 22:10:18.912455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.912435 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:10:18.915549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.915522 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:18.915621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.915581 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:18.915667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:18.915626 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:19.468724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.468685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" event={"ID":"957524f0-a0ac-49fa-9dfd-50ff2fa43a52","Type":"ContainerStarted","Data":"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff"} Oct 29 22:10:19.469209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.468748 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" event={"ID":"957524f0-a0ac-49fa-9dfd-50ff2fa43a52","Type":"ContainerStarted","Data":"3355540f6b57c58b336307c4b0dd4e669855bc1d971c917278a8beb3f7175d29"} Oct 29 22:10:19.470470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.470443 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" event={"ID":"699c48b1-116c-4902-9025-0505203814fd","Type":"ContainerStarted","Data":"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049"} Oct 29 22:10:19.470578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.470491 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" event={"ID":"699c48b1-116c-4902-9025-0505203814fd","Type":"ContainerStarted","Data":"db42a11bfcaecc3823292dd2defb4679661cd76ae5cfc1e86c0b0c5dba6acc74"} Oct 29 22:10:19.518605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.518528 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podStartSLOduration=1.518511058 podStartE2EDuration="1.518511058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:19.516680316 +0000 UTC m=+1681.258610993" watchObservedRunningTime="2026-10-29 22:10:19.518511058 +0000 UTC m=+1681.260441734" Oct 29 22:10:19.562150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.562129 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:19.563289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.563244 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:19.563562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.563316 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:19.568216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.568174 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podStartSLOduration=1.5681635090000001 podStartE2EDuration="1.568163509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:19.563379898 +0000 UTC m=+1681.305310583" watchObservedRunningTime="2026-10-29 22:10:19.568163509 +0000 UTC m=+1681.310094182" Oct 29 22:10:19.577817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.577796 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:19.578947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.578928 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:19.579046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.578983 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:19.854165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.851356 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s"] Oct 29 22:10:19.897657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.897618 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s"] Oct 29 22:10:19.897848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.897760 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:19.915027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.914999 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:10:19.915209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.915066 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:10:19.915209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.915146 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:10:19.915209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.915152 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:10:19.968103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.951619 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:19.968103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.951704 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:19.968103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:19.951912 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp4q\" (UniqueName: \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.073098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.072550 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.073098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.072641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.073098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.072854 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp4q\" (UniqueName: \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.083098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.074182 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.085061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.084846 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.100296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.100262 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp4q\" (UniqueName: \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\") pod \"maas-api-revoke-sub-f4fb0eaa-p9g5s\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.217779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.217750 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:20.691154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.687593 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:20.691154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.687653 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:20.727097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.698386 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:20.727097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.698445 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:20.828009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.820255 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s"] Oct 29 22:10:20.828009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.827894 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:10:20.844328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:20.839261 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" containerID="cri-o://53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44" gracePeriod=30 Oct 29 22:10:21.116715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.116683 4543 scope.go:122] "RemoveContainer" containerID="572ace117f3789d4379d4131ecfdac5a450ecd4f1c907b553f4e2255956e1b9a" Oct 29 22:10:21.275199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.275171 4543 scope.go:122] "RemoveContainer" containerID="6ee7e9a0d4f913e65dc520c8bfed6d2aec7a93b3d30a4ff2c63a3dc099e14290" Oct 29 22:10:21.356763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.339504 4543 scope.go:122] "RemoveContainer" containerID="20ee25a42b92abcc0f544612e1aebb1c61a9dfe4853599d7519ea1ffac003b41" Oct 29 22:10:21.408276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.408090 4543 scope.go:122] "RemoveContainer" containerID="b1e410f9353b9554e91490d5d53ccbe9838420a6f7907a04a8d82dea0c32b739" Oct 29 22:10:21.507988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.507962 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" event={"ID":"6d8be25b-e7fb-4856-8949-6d56c5d70ea3","Type":"ContainerStarted","Data":"149e5bd441f1bd1588b1d3652755addf50918bf49e039ef15a4f639b9844f221"} Oct 29 22:10:21.574228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.574150 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:21.574466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.574439 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:21.579219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.579199 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:21.579388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.579371 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:21.685829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.685801 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7"] Oct 29 22:10:21.695180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.695158 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7"] Oct 29 22:10:21.695612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.695285 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.883805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.883770 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.884115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.884093 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.884334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.884317 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmvl\" (UniqueName: \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.986245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.986183 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.986696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.986671 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.987016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.986984 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmvl\" (UniqueName: \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.987276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.987034 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:21.989319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:21.989295 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:22.014859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.014839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmvl\" (UniqueName: \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\") pod \"maas-api-revoke-sub-8674edd5-nj2c7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:22.310178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.310141 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:22.589107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.588596 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:22.589107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.588653 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:22.591790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.591652 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:22.591790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.591714 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:22.594766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.594407 4543 generic.go:356] "Generic (PLEG): container finished" podID="6d8be25b-e7fb-4856-8949-6d56c5d70ea3" containerID="8da5d1cdf55ce49e34e65a3af8bceeead58a7c561408200bf6ec874281247cbf" exitCode=0 Oct 29 22:10:22.594766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.594492 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" event={"ID":"6d8be25b-e7fb-4856-8949-6d56c5d70ea3","Type":"ContainerDied","Data":"8da5d1cdf55ce49e34e65a3af8bceeead58a7c561408200bf6ec874281247cbf"} Oct 29 22:10:22.801025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:22.800937 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7"] Oct 29 22:10:22.804009 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:22.803972 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3800f168_65f3_4075_a48a_ac59cbde64a7.slice/crio-fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78 WatchSource:0}: Error finding container fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78: Status 404 returned error can't find the container with id fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78 Oct 29 22:10:23.565099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.563267 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:23.565099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.563330 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:23.589105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.582808 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:23.589105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.582873 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:23.605334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.605296 4543 generic.go:356] "Generic (PLEG): container finished" podID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerID="576001ca5abdc1550ddbeba7caf3dff34d24163e7c0054a9752cb007a6a44ef1" exitCode=0 Oct 29 22:10:23.605527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.605369 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" event={"ID":"a6eb53b7-b336-4de1-95c4-035e94e3ab17","Type":"ContainerDied","Data":"576001ca5abdc1550ddbeba7caf3dff34d24163e7c0054a9752cb007a6a44ef1"} Oct 29 22:10:23.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.607593 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" event={"ID":"3800f168-65f3-4075-a48a-ac59cbde64a7","Type":"ContainerStarted","Data":"1c1861359c7fe68f7cfa210e017fc646beb3c6db7a64f00166a211b15a80f0c2"} Oct 29 22:10:23.607671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.607637 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" event={"ID":"3800f168-65f3-4075-a48a-ac59cbde64a7","Type":"ContainerStarted","Data":"fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78"} Oct 29 22:10:23.699192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:23.690580 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" podStartSLOduration=2.690564997 podStartE2EDuration="2.690564997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:23.684936222 +0000 UTC m=+1685.426866897" watchObservedRunningTime="2026-10-29 22:10:23.690564997 +0000 UTC m=+1685.432495669" Oct 29 22:10:24.020040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.020005 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 22:10:24.119013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.118980 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.119371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.119349 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.119696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.119667 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:24.119788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.119694 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:24.119865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.119681 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.120008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.119994 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.120184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120166 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.120565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120543 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.120794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120775 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.120993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120975 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.121134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.121119 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\") pod \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\" (UID: \"a6eb53b7-b336-4de1-95c4-035e94e3ab17\") " Oct 29 22:10:24.122750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.122729 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.122882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.122869 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.123312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120004 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:24.123414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120310 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:24.123414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.120871 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:24.124752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.124726 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:24.126025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.125999 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "kube-api-access-nmbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:24.126172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.126132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:10:24.209932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.209557 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token" pod "a6eb53b7-b336-4de1-95c4-035e94e3ab17" (UID: "a6eb53b7-b336-4de1-95c4-035e94e3ab17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:24.224847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224806 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224850 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbw2\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-kube-api-access-nmbw2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224872 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224885 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224898 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224910 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb53b7-b336-4de1-95c4-035e94e3ab17-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.225009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.224921 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb53b7-b336-4de1-95c4-035e94e3ab17-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.322823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.322785 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:24.453100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.446539 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\") pod \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " Oct 29 22:10:24.453100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.446777 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\") pod \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " Oct 29 22:10:24.453100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.446918 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\" (UniqueName: \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\") pod \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\" (UID: \"6d8be25b-e7fb-4856-8949-6d56c5d70ea3\") " Oct 29 22:10:24.461144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.460586 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca" pod "6d8be25b-e7fb-4856-8949-6d56c5d70ea3" (UID: "6d8be25b-e7fb-4856-8949-6d56c5d70ea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:24.482097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.466345 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.485520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.485491 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca" pod "6d8be25b-e7fb-4856-8949-6d56c5d70ea3" (UID: "6d8be25b-e7fb-4856-8949-6d56c5d70ea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:24.485836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.485817 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q" pod "6d8be25b-e7fb-4856-8949-6d56c5d70ea3" (UID: "6d8be25b-e7fb-4856-8949-6d56c5d70ea3"). InnerVolumeSpecName "kube-api-access-ztp4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:24.562099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.561446 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:24.562099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.561501 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:24.579495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.579452 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:24.579662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.579510 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:24.581013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.580982 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztp4q\" (UniqueName: \"kubernetes.io/projected/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-kube-api-access-ztp4q\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.581136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.581018 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8be25b-e7fb-4856-8949-6d56c5d70ea3-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:24.617642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.617617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" event={"ID":"a6eb53b7-b336-4de1-95c4-035e94e3ab17","Type":"ContainerDied","Data":"cb03c66e76eb9d3b249ff8a1f059b97abccb483a0132b0cbc66bad43ca35dd6a"} Oct 29 22:10:24.617740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.617671 4543 scope.go:122] "RemoveContainer" containerID="576001ca5abdc1550ddbeba7caf3dff34d24163e7c0054a9752cb007a6a44ef1" Oct 29 22:10:24.617803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.617792 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml" Oct 29 22:10:24.620299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.620234 4543 generic.go:356] "Generic (PLEG): container finished" podID="3800f168-65f3-4075-a48a-ac59cbde64a7" containerID="1c1861359c7fe68f7cfa210e017fc646beb3c6db7a64f00166a211b15a80f0c2" exitCode=0 Oct 29 22:10:24.650112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.648463 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" event={"ID":"3800f168-65f3-4075-a48a-ac59cbde64a7","Type":"ContainerDied","Data":"1c1861359c7fe68f7cfa210e017fc646beb3c6db7a64f00166a211b15a80f0c2"} Oct 29 22:10:24.801525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.801486 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" event={"ID":"6d8be25b-e7fb-4856-8949-6d56c5d70ea3","Type":"ContainerDied","Data":"149e5bd441f1bd1588b1d3652755addf50918bf49e039ef15a4f639b9844f221"} Oct 29 22:10:24.801831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.801812 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149e5bd441f1bd1588b1d3652755addf50918bf49e039ef15a4f639b9844f221" Oct 29 22:10:24.801974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.801526 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s" Oct 29 22:10:24.819385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.819360 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 22:10:24.826516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:24.826491 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-zvbml"] Oct 29 22:10:25.424736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.424667 4543 patch_prober.go:29] interesting pod/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:10:25.424736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.424728 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:10:25.561939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.561905 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 22:10:25.562135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.561967 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 22:10:25.577420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.577387 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:25.577578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.577442 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:25.983123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:25.983059 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:26.112461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.110718 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\") pod \"3800f168-65f3-4075-a48a-ac59cbde64a7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " Oct 29 22:10:26.112461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.110799 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\" (UniqueName: \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\") pod \"3800f168-65f3-4075-a48a-ac59cbde64a7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " Oct 29 22:10:26.112461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.110999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\") pod \"3800f168-65f3-4075-a48a-ac59cbde64a7\" (UID: \"3800f168-65f3-4075-a48a-ac59cbde64a7\") " Oct 29 22:10:26.112461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.111449 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca" pod "3800f168-65f3-4075-a48a-ac59cbde64a7" (UID: "3800f168-65f3-4075-a48a-ac59cbde64a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:26.112461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.112417 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.116593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.116566 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca" pod "3800f168-65f3-4075-a48a-ac59cbde64a7" (UID: "3800f168-65f3-4075-a48a-ac59cbde64a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:26.116754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.116714 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl" pod "3800f168-65f3-4075-a48a-ac59cbde64a7" (UID: "3800f168-65f3-4075-a48a-ac59cbde64a7"). InnerVolumeSpecName "kube-api-access-7zmvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:26.216881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.214009 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3800f168-65f3-4075-a48a-ac59cbde64a7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.216881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.214051 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmvl\" (UniqueName: \"kubernetes.io/projected/3800f168-65f3-4075-a48a-ac59cbde64a7-kube-api-access-7zmvl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.326034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.326008 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:10:26.395647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.395572 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" path="/var/lib/kubelet/pods/a6eb53b7-b336-4de1-95c4-035e94e3ab17/volumes" Oct 29 22:10:26.416776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.416751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.416894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.416792 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.416894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.416853 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.416935 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.416969 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417111 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417151 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417192 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417346 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\") pod \"eb57b573-9951-4cb4-b668-a352bcbf0936\" (UID: \"eb57b573-9951-4cb4-b668-a352bcbf0936\") " Oct 29 22:10:26.417559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417531 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:26.417769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:26.417769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:26.417975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.417952 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:26.418244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.418219 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.418328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.418256 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.418328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.418270 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.418328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.418286 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.418328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.418224 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:26.420365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.420341 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "kube-api-access-nrt58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:26.420543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.420521 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:10:26.420608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.420526 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:26.420660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.420636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo" pod "eb57b573-9951-4cb4-b668-a352bcbf0936" (UID: "eb57b573-9951-4cb4-b668-a352bcbf0936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:10:26.519753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.519562 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.519753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.519592 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrt58\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-kube-api-access-nrt58\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.519753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.519606 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.519753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.519621 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb57b573-9951-4cb4-b668-a352bcbf0936-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.519753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.519634 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb57b573-9951-4cb4-b668-a352bcbf0936-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:26.577758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.577718 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 29 22:10:26.578024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.577798 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 29 22:10:26.829494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.829454 4543 generic.go:356] "Generic (PLEG): container finished" podID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerID="53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44" exitCode=0 Oct 29 22:10:26.829708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.829603 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerDied","Data":"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44"} Oct 29 22:10:26.829708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.829657 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" event={"ID":"eb57b573-9951-4cb4-b668-a352bcbf0936","Type":"ContainerDied","Data":"43e057b601df2521a9b867a0288a40717815f3169c328be2a0fd8785d04d618e"} Oct 29 22:10:26.829708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.829692 4543 scope.go:122] "RemoveContainer" containerID="53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44" Oct 29 22:10:26.829907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.829825 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr" Oct 29 22:10:26.838022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.837988 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" event={"ID":"3800f168-65f3-4075-a48a-ac59cbde64a7","Type":"ContainerDied","Data":"fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78"} Oct 29 22:10:26.838205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.838048 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb33443a123601ddc50913f08472f41604eb7a0855923a0a019ab6a721dcab78" Oct 29 22:10:26.838205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.838134 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7" Oct 29 22:10:26.850872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.850851 4543 scope.go:122] "RemoveContainer" containerID="a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434" Oct 29 22:10:26.867415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.867392 4543 scope.go:122] "RemoveContainer" containerID="53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44" Oct 29 22:10:26.867754 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:10:26.867728 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44\": container with ID starting with 53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44 not found: ID does not exist" containerID="53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44" Oct 29 22:10:26.867962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.867929 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44"} err="failed to get container status \"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44\": rpc error: code = NotFound desc = could not find container \"53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44\": container with ID starting with 53d90af5287b7d491d71086697cf23c8f639965622fe927e5d248316897e8c44 not found: ID does not exist" Oct 29 22:10:26.868135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.867962 4543 scope.go:122] "RemoveContainer" containerID="a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434" Oct 29 22:10:26.868285 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:10:26.868261 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434\": container with ID starting with a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434 not found: ID does not exist" containerID="a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434" Oct 29 22:10:26.868454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.868419 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434"} err="failed to get container status \"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434\": rpc error: code = NotFound desc = could not find container \"a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434\": container with ID starting with a7c0480e25a0a7b68352eb81c4f486f1127c4280f2352fa3a6d6e3f4c6c0d434 not found: ID does not exist" Oct 29 22:10:26.910273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.910248 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:10:26.919051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:26.919024 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c53cc8-openshift-default-849bcf445f-llvzr"] Oct 29 22:10:27.096043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.095952 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:27.096448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.096424 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:27.097334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.097312 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:10:27.582539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.582511 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:27.583010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.582996 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:27.583686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:27.583667 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:10:27.961410 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:10:27.961385 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:10:28.394857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:28.394772 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" path="/var/lib/kubelet/pods/eb57b573-9951-4cb4-b668-a352bcbf0936/volumes" Oct 29 22:10:29.453732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.453703 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:10:29.454382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454367 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6d8be25b-e7fb-4856-8949-6d56c5d70ea3" containerName="revoke-keys" Oct 29 22:10:29.454427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454384 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8be25b-e7fb-4856-8949-6d56c5d70ea3" containerName="revoke-keys" Oct 29 22:10:29.454427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454397 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3800f168-65f3-4075-a48a-ac59cbde64a7" containerName="revoke-keys" Oct 29 22:10:29.454427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454404 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3800f168-65f3-4075-a48a-ac59cbde64a7" containerName="revoke-keys" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454448 4543 cpu_manager.go:414] "containerMap: removing container" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454454 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454463 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454467 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454478 4543 cpu_manager.go:414] "containerMap: removing container" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454483 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454578 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454586 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb53b7-b336-4de1-95c4-035e94e3ab17" containerName="istio-proxy" Oct 29 22:10:29.454706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454591 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb57b573-9951-4cb4-b668-a352bcbf0936" containerName="istio-proxy" Oct 29 22:10:29.454706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454597 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8be25b-e7fb-4856-8949-6d56c5d70ea3" containerName="revoke-keys" Oct 29 22:10:29.454706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.454603 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3800f168-65f3-4075-a48a-ac59cbde64a7" containerName="revoke-keys" Oct 29 22:10:29.459892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.459875 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.481931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.481903 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:10:29.490013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.489993 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7721b3\"" Oct 29 22:10:29.490013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.490003 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7721b3-dockercfg-rv67z\"" Oct 29 22:10:29.626981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.626813 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.627850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.627827 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945tz\" (UniqueName: \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.628917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.628901 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:10:29.644643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.644623 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:10:29.644887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.644842 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.732829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.731102 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945tz\" (UniqueName: \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.732829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.731213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.732829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.731266 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4jx\" (UniqueName: \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.732829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.731350 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.732829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.732645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.776863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.776827 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945tz\" (UniqueName: \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\") pod \"payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:29.832677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.832647 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.832837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.832699 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4jx\" (UniqueName: \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.833438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.833419 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.859813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.859779 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4jx\" (UniqueName: \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\") pod \"payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:29.984022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:29.983953 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:30.075468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.075439 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:30.298058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.298035 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:10:30.298855 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:30.298828 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafde8e3_a87c_42dc_9189_e9a8a1043a40.slice/crio-8d908901378b0e1b2e52a73a6af27fede1f2857862c1f97ed666bbcabfa8cad7 WatchSource:0}: Error finding container 8d908901378b0e1b2e52a73a6af27fede1f2857862c1f97ed666bbcabfa8cad7: Status 404 returned error can't find the container with id 8d908901378b0e1b2e52a73a6af27fede1f2857862c1f97ed666bbcabfa8cad7 Oct 29 22:10:30.412092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.412047 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:10:30.412215 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:30.412195 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9982b2b_afc7_4802_aebb_98b3dba4d46b.slice/crio-c7d9c291cf2b4734a3b9e98a98e9329489d30d782b9fcb554c42c54aa65867ac WatchSource:0}: Error finding container c7d9c291cf2b4734a3b9e98a98e9329489d30d782b9fcb554c42c54aa65867ac: Status 404 returned error can't find the container with id c7d9c291cf2b4734a3b9e98a98e9329489d30d782b9fcb554c42c54aa65867ac Oct 29 22:10:30.874135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.874105 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" event={"ID":"e9982b2b-afc7-4802-aebb-98b3dba4d46b","Type":"ContainerStarted","Data":"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b"} Oct 29 22:10:30.874550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.874166 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" event={"ID":"e9982b2b-afc7-4802-aebb-98b3dba4d46b","Type":"ContainerStarted","Data":"c7d9c291cf2b4734a3b9e98a98e9329489d30d782b9fcb554c42c54aa65867ac"} Oct 29 22:10:30.874550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.874197 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:30.875844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.875817 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" event={"ID":"dafde8e3-a87c-42dc-9189-e9a8a1043a40","Type":"ContainerStarted","Data":"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159"} Oct 29 22:10:30.876067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.875868 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" event={"ID":"dafde8e3-a87c-42dc-9189-e9a8a1043a40","Type":"ContainerStarted","Data":"8d908901378b0e1b2e52a73a6af27fede1f2857862c1f97ed666bbcabfa8cad7"} Oct 29 22:10:30.876067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.875896 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:30.918497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.918451 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" podStartSLOduration=1.918439114 podStartE2EDuration="1.918439114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.916780827 +0000 UTC m=+1692.658711501" watchObservedRunningTime="2026-10-29 22:10:30.918439114 +0000 UTC m=+1692.660369812" Oct 29 22:10:30.965821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:30.965776 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" podStartSLOduration=1.965764401 podStartE2EDuration="1.965764401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.961370097 +0000 UTC m=+1692.703300770" watchObservedRunningTime="2026-10-29 22:10:30.965764401 +0000 UTC m=+1692.707695108" Oct 29 22:10:32.497218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.497186 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l"] Oct 29 22:10:32.502866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.502848 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.521604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.521578 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l"] Oct 29 22:10:32.530042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.530008 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:10:32.530207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.530051 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:10:32.530207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.530066 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:10:32.538987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.538970 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:10:32.564853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.564810 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.564982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.564888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.565055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.565013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz8b\" (UniqueName: \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.665842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.665822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.665924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.665898 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz8b\" (UniqueName: \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.666040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.666030 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.666653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.666590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.668343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.668324 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.692270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.692252 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz8b\" (UniqueName: \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\") pod \"maas-api-revoke-sub-877eed8c-c9r8l\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:32.817860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:32.817812 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:33.129709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:33.129670 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l"] Oct 29 22:10:33.129870 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:33.129688 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3912a28_cdd8_4299_ab7a_0ca2f63b1075.slice/crio-ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd WatchSource:0}: Error finding container ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd: Status 404 returned error can't find the container with id ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd Oct 29 22:10:33.905434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:33.905400 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" event={"ID":"a3912a28-cdd8-4299-ab7a-0ca2f63b1075","Type":"ContainerStarted","Data":"e5659a79370489909f5ef46f3f580bc772217610c1acf0f9c4c72ed53ae35409"} Oct 29 22:10:33.905782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:33.905451 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" event={"ID":"a3912a28-cdd8-4299-ab7a-0ca2f63b1075","Type":"ContainerStarted","Data":"ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd"} Oct 29 22:10:33.951347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:33.951306 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" podStartSLOduration=1.951292788 podStartE2EDuration="1.951292788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:33.94918457 +0000 UTC m=+1695.691115242" watchObservedRunningTime="2026-10-29 22:10:33.951292788 +0000 UTC m=+1695.693223460" Oct 29 22:10:34.914349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:34.914318 4543 generic.go:356] "Generic (PLEG): container finished" podID="a3912a28-cdd8-4299-ab7a-0ca2f63b1075" containerID="e5659a79370489909f5ef46f3f580bc772217610c1acf0f9c4c72ed53ae35409" exitCode=0 Oct 29 22:10:34.914730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:34.914407 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" event={"ID":"a3912a28-cdd8-4299-ab7a-0ca2f63b1075","Type":"ContainerDied","Data":"e5659a79370489909f5ef46f3f580bc772217610c1acf0f9c4c72ed53ae35409"} Oct 29 22:10:35.932474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:35.932449 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq"] Oct 29 22:10:35.936986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:35.936964 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:35.959898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:35.959871 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq"] Oct 29 22:10:36.008293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.008256 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.008554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.008534 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhnx\" (UniqueName: \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.008619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.008581 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.109895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.109868 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhnx\" (UniqueName: \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.110023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.109920 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.110108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.110039 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.110680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.110658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.112321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.112300 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.113481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.113463 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:36.135614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.135591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhnx\" (UniqueName: \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\") pod \"maas-api-revoke-sub-804585dd-mlgcq\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.211194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.211173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\") pod \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " Oct 29 22:10:36.211362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.211344 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\" (UniqueName: \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\") pod \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " Oct 29 22:10:36.211472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.211454 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\") pod \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\" (UID: \"a3912a28-cdd8-4299-ab7a-0ca2f63b1075\") " Oct 29 22:10:36.211566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.211544 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca" pod "a3912a28-cdd8-4299-ab7a-0ca2f63b1075" (UID: "a3912a28-cdd8-4299-ab7a-0ca2f63b1075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:36.212239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.212219 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:36.213587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.213562 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca" pod "a3912a28-cdd8-4299-ab7a-0ca2f63b1075" (UID: "a3912a28-cdd8-4299-ab7a-0ca2f63b1075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:36.213689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.213612 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b" pod "a3912a28-cdd8-4299-ab7a-0ca2f63b1075" (UID: "a3912a28-cdd8-4299-ab7a-0ca2f63b1075"). InnerVolumeSpecName "kube-api-access-jvz8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:36.256857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.256832 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:36.314239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.314206 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvz8b\" (UniqueName: \"kubernetes.io/projected/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-kube-api-access-jvz8b\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:36.314239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.314238 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3912a28-cdd8-4299-ab7a-0ca2f63b1075-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:36.570001 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:36.569975 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca4b533_1e91_470e_964e_4ad8e5cc15f5.slice/crio-2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa WatchSource:0}: Error finding container 2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa: Status 404 returned error can't find the container with id 2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa Oct 29 22:10:36.571436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.571412 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq"] Oct 29 22:10:36.931642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.931612 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" event={"ID":"8ca4b533-1e91-470e-964e-4ad8e5cc15f5","Type":"ContainerStarted","Data":"e09585457b278baf15bb5aa2921ef0fd6fb44a23607cfc2a1e18a5582632c288"} Oct 29 22:10:36.931772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.931664 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" event={"ID":"8ca4b533-1e91-470e-964e-4ad8e5cc15f5","Type":"ContainerStarted","Data":"2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa"} Oct 29 22:10:36.933151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.933127 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" Oct 29 22:10:36.933531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.933130 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l" event={"ID":"a3912a28-cdd8-4299-ab7a-0ca2f63b1075","Type":"ContainerDied","Data":"ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd"} Oct 29 22:10:36.933531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.933280 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce4c1b034fef5235bd23b0a7b62c3e22feb0b88da72500d9c82b052e3ff345cd" Oct 29 22:10:36.976089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:36.976024 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" podStartSLOduration=1.976009011 podStartE2EDuration="1.976009011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:36.975274159 +0000 UTC m=+1698.717204836" watchObservedRunningTime="2026-10-29 22:10:36.976009011 +0000 UTC m=+1698.717939684" Oct 29 22:10:37.944331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:37.944288 4543 generic.go:356] "Generic (PLEG): container finished" podID="8ca4b533-1e91-470e-964e-4ad8e5cc15f5" containerID="e09585457b278baf15bb5aa2921ef0fd6fb44a23607cfc2a1e18a5582632c288" exitCode=0 Oct 29 22:10:37.944685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:37.944360 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" event={"ID":"8ca4b533-1e91-470e-964e-4ad8e5cc15f5","Type":"ContainerDied","Data":"e09585457b278baf15bb5aa2921ef0fd6fb44a23607cfc2a1e18a5582632c288"} Oct 29 22:10:39.133737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.133712 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:39.249507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.249479 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\") pod \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " Oct 29 22:10:39.249640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.249557 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\" (UniqueName: \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\") pod \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " Oct 29 22:10:39.249732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.249720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\") pod \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\" (UID: \"8ca4b533-1e91-470e-964e-4ad8e5cc15f5\") " Oct 29 22:10:39.249814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.249792 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca" pod "8ca4b533-1e91-470e-964e-4ad8e5cc15f5" (UID: "8ca4b533-1e91-470e-964e-4ad8e5cc15f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:39.250115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.250069 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:39.251805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.251784 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx" pod "8ca4b533-1e91-470e-964e-4ad8e5cc15f5" (UID: "8ca4b533-1e91-470e-964e-4ad8e5cc15f5"). InnerVolumeSpecName "kube-api-access-rxhnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:39.251898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.251810 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca" pod "8ca4b533-1e91-470e-964e-4ad8e5cc15f5" (UID: "8ca4b533-1e91-470e-964e-4ad8e5cc15f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:39.351454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.351433 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:39.351454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.351455 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhnx\" (UniqueName: \"kubernetes.io/projected/8ca4b533-1e91-470e-964e-4ad8e5cc15f5-kube-api-access-rxhnx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:39.962992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.962959 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" Oct 29 22:10:39.963194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.963012 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq" event={"ID":"8ca4b533-1e91-470e-964e-4ad8e5cc15f5","Type":"ContainerDied","Data":"2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa"} Oct 29 22:10:39.963194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:39.963067 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b44d4a8a6fd6016328a44cacbaa4e66fb2d6468d382fb0945fb331f28afb1aa" Oct 29 22:10:41.886245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:41.886216 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:10:41.886633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:41.886290 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:10:42.050928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.050901 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:10:42.051654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051638 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a3912a28-cdd8-4299-ab7a-0ca2f63b1075" containerName="revoke-keys" Oct 29 22:10:42.051697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051656 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3912a28-cdd8-4299-ab7a-0ca2f63b1075" containerName="revoke-keys" Oct 29 22:10:42.051697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051672 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8ca4b533-1e91-470e-964e-4ad8e5cc15f5" containerName="revoke-keys" Oct 29 22:10:42.051697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051678 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca4b533-1e91-470e-964e-4ad8e5cc15f5" containerName="revoke-keys" Oct 29 22:10:42.051808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051801 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca4b533-1e91-470e-964e-4ad8e5cc15f5" containerName="revoke-keys" Oct 29 22:10:42.051844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.051820 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3912a28-cdd8-4299-ab7a-0ca2f63b1075" containerName="revoke-keys" Oct 29 22:10:42.055678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.055664 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.079584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.079553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:10:42.207593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xg4\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207577 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207697 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207755 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207843 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.207934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.207921 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.208027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.208014 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.308973 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309007 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309053 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309136 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309179 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xg4\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309213 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309237 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309379 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309480 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309519 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.309886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.309865 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.311660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.311639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.311783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.311766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.334597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.334563 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.334685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.334633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xg4\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.369318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.369296 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:42.699289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.699262 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:10:42.699415 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:42.699258 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f06d6e_8baa_46c1_b142_0ce3ebafee1a.slice/crio-f1d05fcff78e8b80e766ea848afc3723c315b18a92261255a0d180c7b8607bf3 WatchSource:0}: Error finding container f1d05fcff78e8b80e766ea848afc3723c315b18a92261255a0d180c7b8607bf3: Status 404 returned error can't find the container with id f1d05fcff78e8b80e766ea848afc3723c315b18a92261255a0d180c7b8607bf3 Oct 29 22:10:42.701872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.701838 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:42.701963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.701903 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:42.701963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.701934 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:10:42.997635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.997606 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" event={"ID":"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a","Type":"ContainerStarted","Data":"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6"} Oct 29 22:10:42.998018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:42.997657 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" event={"ID":"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a","Type":"ContainerStarted","Data":"f1d05fcff78e8b80e766ea848afc3723c315b18a92261255a0d180c7b8607bf3"} Oct 29 22:10:43.369948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:43.369866 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:43.371900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:43.371872 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:43.372025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:43.371918 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:44.370588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:44.370549 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:44.371054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:44.370610 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:45.370383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:45.370347 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:45.370619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:45.370402 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:46.371099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:46.370269 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:46.371099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:46.370331 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:47.370269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:47.370235 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:47.370435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:47.370290 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:48.376408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:48.376376 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:48.376991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:48.376960 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:49.152107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.127426 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:49.393097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.392373 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:49.393097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.392431 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:49.425383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.417415 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:10:49.516619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.516574 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:49.756197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.731787 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:49.768941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:49.768893 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.133150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.128286 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:10:50.263101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.259424 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.389160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.388213 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:50.389160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.388302 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:50.434108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.422367 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:10:50.519102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.518206 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 29 22:10:50.533113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.529250 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:10:50.533113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.529321 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:10:50.582624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.582564 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podStartSLOduration=8.582544807 podStartE2EDuration="8.582544807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:43.066992378 +0000 UTC m=+1704.808923051" watchObservedRunningTime="2026-10-29 22:10:50.582544807 +0000 UTC m=+1712.324475482" Oct 29 22:10:50.735163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.734227 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.735163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.735024 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:10:50.778125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.772611 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:50.935104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.789170 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:10:50.960274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:50.814896 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.143595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.143476 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:10:51.145111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.145059 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.145940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.145346 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:10:51.205106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.195866 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.376497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.373032 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.433786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.423916 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:51.433786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.423989 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:51.433786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.432862 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.536103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.531192 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 29 22:10:51.746218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.746167 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.835249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:51.835214 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:10:52.256159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.176909 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.279466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.212066 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:10:52.280414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.280389 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:10:52.372460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.372426 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Oct 29 22:10:52.372778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.372735 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Oct 29 22:10:52.373985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.373950 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.374562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.374541 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:10:52.453203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.453157 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.565167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.555374 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 29 22:10:52.740696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.740193 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:10:52.740696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.740263 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:53.011144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.966262 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:10:53.011144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:52.996196 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:53.148172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.148120 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:10:53.286147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.285285 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:10:53.286147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.285729 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:10:53.287647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.287622 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:10:53.297949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.297928 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:10:53.310126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.306979 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:10:53.310126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.307226 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:10:53.310644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.310343 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:10:53.311661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.311585 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:10:53.517619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.517581 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:53.518454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.518429 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:53.519947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:53.519921 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:10:54.367222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.367192 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v"] Oct 29 22:10:54.388848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.388811 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.403688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.403285 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v"] Oct 29 22:10:54.406736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.406708 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:10:54.408789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.408753 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.409067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.409046 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7vgm\" (UniqueName: \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.409237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.409221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.419155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.419134 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:10:54.419268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.419165 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:10:54.419268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.419199 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:10:54.510232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.510197 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.510411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.510316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7vgm\" (UniqueName: \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.510411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.510399 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.510910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.510887 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.512959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.512941 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.570460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.570333 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7vgm\" (UniqueName: \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\") pod \"maas-api-revoke-sub-8c7e49bc-6vc4v\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.664157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.640367 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld"] Oct 29 22:10:54.674232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.674201 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.674423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.674065 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld"] Oct 29 22:10:54.712339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.712315 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:54.712775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.712755 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.712851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.712837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7z48\" (UniqueName: \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.712953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.712924 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.814551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.814511 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7z48\" (UniqueName: \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.814742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.814609 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.814885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.814866 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.816140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.816112 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.823112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.820589 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.915340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.903988 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7z48\" (UniqueName: \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\") pod \"maas-api-revoke-sub-14d3b57c-9gnld\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:54.995583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:54.995323 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:55.127107 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:55.127058 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94014a6d_f044_4a37_ad88_61ff872ec68c.slice/crio-6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650 WatchSource:0}: Error finding container 6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650: Status 404 returned error can't find the container with id 6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650 Oct 29 22:10:55.159352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.156406 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v"] Oct 29 22:10:55.365848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.365806 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld"] Oct 29 22:10:55.369951 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:55.369924 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc90f2df_f022_4ad1_b5d6_d1dca22c7ebb.slice/crio-a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7 WatchSource:0}: Error finding container a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7: Status 404 returned error can't find the container with id a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7 Oct 29 22:10:55.886137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.884245 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" event={"ID":"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb","Type":"ContainerStarted","Data":"4bd6a554a3f3f4df325c3be0a01b48e0f9aa7e9c77327cc2d821163b165876d5"} Oct 29 22:10:55.886137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.884358 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" event={"ID":"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb","Type":"ContainerStarted","Data":"a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7"} Oct 29 22:10:55.889598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.889568 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" event={"ID":"94014a6d-f044-4a37-ad88-61ff872ec68c","Type":"ContainerStarted","Data":"b09f7cf11430ea9f623187db110de0d57632fee086b1435c5fbb32e375b1de2f"} Oct 29 22:10:55.889873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.889852 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" event={"ID":"94014a6d-f044-4a37-ad88-61ff872ec68c","Type":"ContainerStarted","Data":"6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650"} Oct 29 22:10:55.944099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.930457 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" podStartSLOduration=1.9304405949999999 podStartE2EDuration="1.930440595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:55.927359058 +0000 UTC m=+1717.669289732" watchObservedRunningTime="2026-10-29 22:10:55.930440595 +0000 UTC m=+1717.672371269" Oct 29 22:10:55.978589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:55.978549 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" podStartSLOduration=1.978535773 podStartE2EDuration="1.978535773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:55.973762326 +0000 UTC m=+1717.715692998" watchObservedRunningTime="2026-10-29 22:10:55.978535773 +0000 UTC m=+1717.720466448" Oct 29 22:10:56.900023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:56.899993 4543 generic.go:356] "Generic (PLEG): container finished" podID="fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" containerID="4bd6a554a3f3f4df325c3be0a01b48e0f9aa7e9c77327cc2d821163b165876d5" exitCode=0 Oct 29 22:10:56.900453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:56.900111 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" event={"ID":"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb","Type":"ContainerDied","Data":"4bd6a554a3f3f4df325c3be0a01b48e0f9aa7e9c77327cc2d821163b165876d5"} Oct 29 22:10:56.901855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:56.901829 4543 generic.go:356] "Generic (PLEG): container finished" podID="94014a6d-f044-4a37-ad88-61ff872ec68c" containerID="b09f7cf11430ea9f623187db110de0d57632fee086b1435c5fbb32e375b1de2f" exitCode=0 Oct 29 22:10:56.902007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:56.901913 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" event={"ID":"94014a6d-f044-4a37-ad88-61ff872ec68c","Type":"ContainerDied","Data":"b09f7cf11430ea9f623187db110de0d57632fee086b1435c5fbb32e375b1de2f"} Oct 29 22:10:57.977691 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:10:57.977645 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:10:58.106180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.106139 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:10:58.112784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.112669 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.131415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.131383 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:10:58.140889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.138437 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-665b09\"" Oct 29 22:10:58.141128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.139231 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-665b09-dockercfg-9sqkw\"" Oct 29 22:10:58.217233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.207516 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.217233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.207777 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dk9\" (UniqueName: \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.248112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.241602 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:10:58.272885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.272511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:58.300063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.300034 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:10:58.300387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.300369 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.313001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.308834 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dk9\" (UniqueName: \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.313001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.309014 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.313001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.310210 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.363002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.362945 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:58.372054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.372021 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dk9\" (UniqueName: \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\") pod \"payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.409954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.409921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\" (UniqueName: \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\") pod \"94014a6d-f044-4a37-ad88-61ff872ec68c\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " Oct 29 22:10:58.410260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.410190 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\") pod \"94014a6d-f044-4a37-ad88-61ff872ec68c\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " Oct 29 22:10:58.410260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.410245 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\") pod \"94014a6d-f044-4a37-ad88-61ff872ec68c\" (UID: \"94014a6d-f044-4a37-ad88-61ff872ec68c\") " Oct 29 22:10:58.410729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.410709 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qj6c\" (UniqueName: \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.410991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.410967 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.412135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.412066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca" pod "94014a6d-f044-4a37-ad88-61ff872ec68c" (UID: "94014a6d-f044-4a37-ad88-61ff872ec68c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:58.412960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.412927 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm" pod "94014a6d-f044-4a37-ad88-61ff872ec68c" (UID: "94014a6d-f044-4a37-ad88-61ff872ec68c"). InnerVolumeSpecName "kube-api-access-h7vgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:58.413038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.412983 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca" pod "94014a6d-f044-4a37-ad88-61ff872ec68c" (UID: "94014a6d-f044-4a37-ad88-61ff872ec68c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:58.512727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.512607 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\") pod \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " Oct 29 22:10:58.512920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.512801 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\") pod \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " Oct 29 22:10:58.512920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.512865 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\" (UniqueName: \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\") pod \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\" (UID: \"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb\") " Oct 29 22:10:58.513591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513208 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca" pod "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" (UID: "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:58.513591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513405 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj6c\" (UniqueName: \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.513796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513676 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.513859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513825 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.513859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513851 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94014a6d-f044-4a37-ad88-61ff872ec68c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.513975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513866 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.513975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.513882 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7vgm\" (UniqueName: \"kubernetes.io/projected/94014a6d-f044-4a37-ad88-61ff872ec68c-kube-api-access-h7vgm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.514758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.514734 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.515673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.515644 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca" pod "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" (UID: "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:58.517642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.517619 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48" pod "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" (UID: "fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb"). InnerVolumeSpecName "kube-api-access-r7z48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:58.539658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.539626 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj6c\" (UniqueName: \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\") pod \"payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.615943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.615905 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7z48\" (UniqueName: \"kubernetes.io/projected/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-kube-api-access-r7z48\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.615943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.615937 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:10:58.652494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.652460 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:10:58.711705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.711469 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:58.952655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.937611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" event={"ID":"fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb","Type":"ContainerDied","Data":"a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7"} Oct 29 22:10:58.952655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.937715 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a25eec556facef7713c811de41ba1e20d1bf572497db022120dd09a4645b84b7" Oct 29 22:10:58.952655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.937822 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld" Oct 29 22:10:58.959561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.957877 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" Oct 29 22:10:58.959561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.958314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v" event={"ID":"94014a6d-f044-4a37-ad88-61ff872ec68c","Type":"ContainerDied","Data":"6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650"} Oct 29 22:10:58.959561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.958428 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6063556f23ef648e204a1f2b119b94155fedacb6df6d71e7369ea3623153c650" Oct 29 22:10:58.990542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:58.989244 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:10:58.992302 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:58.992273 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa56d71_090c_4ba2_be75_cace1ab71f58.slice/crio-681816e9f9dcb18a69d4f37fecb31665bd19aa18540cdd5cb73808e89247eb67 WatchSource:0}: Error finding container 681816e9f9dcb18a69d4f37fecb31665bd19aa18540cdd5cb73808e89247eb67: Status 404 returned error can't find the container with id 681816e9f9dcb18a69d4f37fecb31665bd19aa18540cdd5cb73808e89247eb67 Oct 29 22:10:59.093376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.093305 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:10:59.097753 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:10:59.097726 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e8e008_e789_4c40_87b6_2c1ee0c5b91a.slice/crio-4b5b962247ede1e03aea34f841f7bf5d2c75ca6686c5ed7ef00027430f92467c WatchSource:0}: Error finding container 4b5b962247ede1e03aea34f841f7bf5d2c75ca6686c5ed7ef00027430f92467c: Status 404 returned error can't find the container with id 4b5b962247ede1e03aea34f841f7bf5d2c75ca6686c5ed7ef00027430f92467c Oct 29 22:10:59.988319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.988277 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" event={"ID":"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a","Type":"ContainerStarted","Data":"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1"} Oct 29 22:10:59.988523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.988339 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" event={"ID":"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a","Type":"ContainerStarted","Data":"4b5b962247ede1e03aea34f841f7bf5d2c75ca6686c5ed7ef00027430f92467c"} Oct 29 22:10:59.988523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.988370 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:10:59.990603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.990561 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" event={"ID":"4aa56d71-090c-4ba2-be75-cace1ab71f58","Type":"ContainerStarted","Data":"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318"} Oct 29 22:10:59.991017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.990606 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" event={"ID":"4aa56d71-090c-4ba2-be75-cace1ab71f58","Type":"ContainerStarted","Data":"681816e9f9dcb18a69d4f37fecb31665bd19aa18540cdd5cb73808e89247eb67"} Oct 29 22:10:59.991017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:10:59.991000 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:11:00.039264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.039215 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" podStartSLOduration=2.039203654 podStartE2EDuration="2.039203654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:00.033231848 +0000 UTC m=+1721.775162523" watchObservedRunningTime="2026-10-29 22:11:00.039203654 +0000 UTC m=+1721.781134327" Oct 29 22:11:00.079889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.079846 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" podStartSLOduration=2.079834004 podStartE2EDuration="2.079834004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:00.077005658 +0000 UTC m=+1721.818936333" watchObservedRunningTime="2026-10-29 22:11:00.079834004 +0000 UTC m=+1721.821764676" Oct 29 22:11:00.276495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.276423 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k"] Oct 29 22:11:00.277291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277211 4543 cpu_manager.go:414] "containerMap: removing container" podUID="94014a6d-f044-4a37-ad88-61ff872ec68c" containerName="revoke-keys" Oct 29 22:11:00.277291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277230 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94014a6d-f044-4a37-ad88-61ff872ec68c" containerName="revoke-keys" Oct 29 22:11:00.277291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277249 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" containerName="revoke-keys" Oct 29 22:11:00.277291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277254 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" containerName="revoke-keys" Oct 29 22:11:00.277530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277375 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="94014a6d-f044-4a37-ad88-61ff872ec68c" containerName="revoke-keys" Oct 29 22:11:00.277530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.277386 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" containerName="revoke-keys" Oct 29 22:11:00.282555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.282534 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.301830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.299921 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:00.301830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.300054 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:00.301830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.300165 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:00.301830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.300210 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:00.301830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.301050 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k"] Oct 29 22:11:00.450410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.450378 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjtm\" (UniqueName: \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.450575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.450441 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.450642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.450614 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.552355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.552280 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.552559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.552540 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjtm\" (UniqueName: \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.552655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.552624 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.553237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.553217 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.554760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.554741 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.578449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.578426 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjtm\" (UniqueName: \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\") pod \"maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.606351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.606332 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:00.943387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:00.943358 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k"] Oct 29 22:11:00.944370 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:00.944338 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba591fe_cc1d_4c42_8aa1_2c9e77dab5db.slice/crio-816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a WatchSource:0}: Error finding container 816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a: Status 404 returned error can't find the container with id 816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a Oct 29 22:11:01.000396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.000371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" event={"ID":"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db","Type":"ContainerStarted","Data":"816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a"} Oct 29 22:11:01.002189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.002153 4543 generic.go:356] "Generic (PLEG): container finished" podID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerID="6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83" exitCode=0 Oct 29 22:11:01.002295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.002196 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerDied","Data":"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83"} Oct 29 22:11:01.002983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.002961 4543 scope.go:122] "RemoveContainer" containerID="6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83" Oct 29 22:11:01.004790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.004759 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:01.004902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.004850 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:01.004902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:01.004885 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:02.012144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.012106 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerStarted","Data":"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507"} Oct 29 22:11:02.013967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.013942 4543 generic.go:356] "Generic (PLEG): container finished" podID="4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" containerID="924c07238de41a7213745be2f7fe559187e0c8824823a1250cadcdb061856a02" exitCode=0 Oct 29 22:11:02.014147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.014005 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" event={"ID":"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db","Type":"ContainerDied","Data":"924c07238de41a7213745be2f7fe559187e0c8824823a1250cadcdb061856a02"} Oct 29 22:11:02.727519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.727482 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:11:02.727748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.727671 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:11:02.727748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:02.727726 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 29 22:11:03.265486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.265464 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:03.300540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.300484 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\") pod \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " Oct 29 22:11:03.300540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.300534 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\") pod \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " Oct 29 22:11:03.300719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.300659 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\" (UniqueName: \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\") pod \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\" (UID: \"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db\") " Oct 29 22:11:03.300930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.300906 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca" pod "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" (UID: "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:03.301349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.301327 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:03.302884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.302849 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca" pod "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" (UID: "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:03.302971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.302930 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm" pod "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" (UID: "4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db"). InnerVolumeSpecName "kube-api-access-rwjtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:03.403136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.403109 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjtm\" (UniqueName: \"kubernetes.io/projected/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-kube-api-access-rwjtm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:03.403136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.403135 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:03.727906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.727875 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:11:03.728134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:03.727927 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 29 22:11:04.055785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:04.055702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" event={"ID":"4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db","Type":"ContainerDied","Data":"816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a"} Oct 29 22:11:04.055785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:04.055750 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816a98b3e78a7c83bec7d2376a702089320c7de33a2c1022cafd7bd329b7022a" Oct 29 22:11:04.055785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:04.055759 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k" Oct 29 22:11:04.727807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:04.727776 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:11:04.728283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:04.727832 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 29 22:11:05.269422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:05.269383 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng"] Oct 29 22:11:05.274262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:05.274232 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fe45b-7fcng"] Oct 29 22:11:05.733101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:05.728118 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:11:05.733101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:05.728175 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 29 22:11:06.386942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:06.386898 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459137e4-1986-441c-b87a-c093dbbd74e0" path="/var/lib/kubelet/pods/459137e4-1986-441c-b87a-c093dbbd74e0/volumes" Oct 29 22:11:06.727353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:06.727327 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:11:06.727515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:06.727377 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 29 22:11:07.353285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.353254 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x"] Oct 29 22:11:07.353969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.353956 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" containerName="revoke-keys" Oct 29 22:11:07.354031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.353970 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" containerName="revoke-keys" Oct 29 22:11:07.354161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.354150 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" containerName="revoke-keys" Oct 29 22:11:07.357749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.357629 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.377805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.377781 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x"] Oct 29 22:11:07.391135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.385330 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:07.391135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.385406 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:07.415009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.414974 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv7r\" (UniqueName: \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.415210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.415113 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.415297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.415275 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.468355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.468332 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:07.468501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.468384 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:07.516639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.516617 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.516879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.516859 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv7r\" (UniqueName: \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.516973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.516941 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.517407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.517388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.519478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.519454 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.542659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.542633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv7r\" (UniqueName: \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\") pod \"maas-api-revoke-sub-53d62f35-l4r2x\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.676102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.676041 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:07.734812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.734775 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:11:07.735970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.735950 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:11:07.736587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.736567 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:11:07.990184 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:07.990155 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c64b6c_87e5_4654_a583_9fe76e42475c.slice/crio-24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b WatchSource:0}: Error finding container 24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b: Status 404 returned error can't find the container with id 24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b Oct 29 22:11:07.990397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:07.990375 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x"] Oct 29 22:11:08.108695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:08.108656 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" event={"ID":"01c64b6c-87e5-4654-a583-9fe76e42475c","Type":"ContainerStarted","Data":"24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b"} Oct 29 22:11:09.122598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:09.122508 4543 generic.go:356] "Generic (PLEG): container finished" podID="01c64b6c-87e5-4654-a583-9fe76e42475c" containerID="10389de19d71d65ca7cd3a8e3fc62606a655c6062b581fcdcaa0fc6fccd290af" exitCode=0 Oct 29 22:11:09.122992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:09.122598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" event={"ID":"01c64b6c-87e5-4654-a583-9fe76e42475c","Type":"ContainerDied","Data":"10389de19d71d65ca7cd3a8e3fc62606a655c6062b581fcdcaa0fc6fccd290af"} Oct 29 22:11:10.290790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.290754 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 22:11:10.291168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.290982 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" containerID="cri-o://ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607" gracePeriod=30 Oct 29 22:11:10.304687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.304664 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:10.351464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.351440 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\") pod \"01c64b6c-87e5-4654-a583-9fe76e42475c\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " Oct 29 22:11:10.351620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.351493 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\") pod \"01c64b6c-87e5-4654-a583-9fe76e42475c\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " Oct 29 22:11:10.351620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.351545 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\" (UniqueName: \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\") pod \"01c64b6c-87e5-4654-a583-9fe76e42475c\" (UID: \"01c64b6c-87e5-4654-a583-9fe76e42475c\") " Oct 29 22:11:10.351859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.351832 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca" pod "01c64b6c-87e5-4654-a583-9fe76e42475c" (UID: "01c64b6c-87e5-4654-a583-9fe76e42475c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:10.352310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.352260 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:10.353689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.353667 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca" pod "01c64b6c-87e5-4654-a583-9fe76e42475c" (UID: "01c64b6c-87e5-4654-a583-9fe76e42475c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:10.353892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.353869 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r" pod "01c64b6c-87e5-4654-a583-9fe76e42475c" (UID: "01c64b6c-87e5-4654-a583-9fe76e42475c"). InnerVolumeSpecName "kube-api-access-cfv7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:10.455361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.455329 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c64b6c-87e5-4654-a583-9fe76e42475c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:10.455361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.455363 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfv7r\" (UniqueName: \"kubernetes.io/projected/01c64b6c-87e5-4654-a583-9fe76e42475c-kube-api-access-cfv7r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:10.801145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.801108 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:11:10.839493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:10.839466 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:11:11.005865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:11.005790 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:11:11.006540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:11.006480 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:11:11.165633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:11.165597 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" event={"ID":"01c64b6c-87e5-4654-a583-9fe76e42475c","Type":"ContainerDied","Data":"24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b"} Oct 29 22:11:11.165826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:11.165645 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x" Oct 29 22:11:11.165826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:11.165657 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ce674f054761dc9d36498a929ee0cd8feb013930e245d2dbebd547185a654b" Oct 29 22:11:15.090216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.090184 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8"] Oct 29 22:11:15.091033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.091017 4543 cpu_manager.go:414] "containerMap: removing container" podUID="01c64b6c-87e5-4654-a583-9fe76e42475c" containerName="revoke-keys" Oct 29 22:11:15.091122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.091035 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c64b6c-87e5-4654-a583-9fe76e42475c" containerName="revoke-keys" Oct 29 22:11:15.091188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.091174 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c64b6c-87e5-4654-a583-9fe76e42475c" containerName="revoke-keys" Oct 29 22:11:15.094213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.094194 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.112952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.112931 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:15.113199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.113186 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:15.113310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.113268 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:15.113310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.113305 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:15.115918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.115895 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8"] Oct 29 22:11:15.221190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.221150 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.221353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.221206 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.221405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.221383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dghb\" (UniqueName: \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.322709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.322683 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dghb\" (UniqueName: \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.322888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.322872 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.322949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.322901 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.323515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.323488 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.325343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.325318 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.348440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.348395 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dghb\" (UniqueName: \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\") pod \"maas-api-revoke-sub-53371b15-skts8\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.408158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.408139 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:15.607123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.607045 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 22:11:15.725731 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:15.725697 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd82c7_d7d9_4e97_88bc_165887719d45.slice/crio-b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef WatchSource:0}: Error finding container b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef: Status 404 returned error can't find the container with id b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef Oct 29 22:11:15.725926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.725908 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8"] Oct 29 22:11:15.728169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728145 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728191 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728308 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728348 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728422 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728557 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728601 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728650 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.728760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:15.728875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728805 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:15.728875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.728844 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\") pod \"e096a529-de3b-465f-8805-c0a68eaf0452\" (UID: \"e096a529-de3b-465f-8805-c0a68eaf0452\") " Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729126 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729304 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729400 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729665 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729689 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729705 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.729718 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096a529-de3b-465f-8805-c0a68eaf0452-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.730681 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.730723 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "kube-api-access-vnqqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:15.731126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.730741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:15.731827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.731726 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo" pod "e096a529-de3b-465f-8805-c0a68eaf0452" (UID: "e096a529-de3b-465f-8805-c0a68eaf0452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:15.801477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.801422 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" containerID="cri-o://33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6" gracePeriod=25 Oct 29 22:11:15.831540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.831516 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.831540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.831539 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096a529-de3b-465f-8805-c0a68eaf0452-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.831692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.831549 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqqd\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-kube-api-access-vnqqd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.831692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.831558 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096a529-de3b-465f-8805-c0a68eaf0452-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.831692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.831567 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096a529-de3b-465f-8805-c0a68eaf0452-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:15.839714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:15.839680 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" containerID="cri-o://f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195" gracePeriod=25 Oct 29 22:11:16.211395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.211366 4543 generic.go:356] "Generic (PLEG): container finished" podID="e096a529-de3b-465f-8805-c0a68eaf0452" containerID="ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607" exitCode=0 Oct 29 22:11:16.211842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.211420 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" Oct 29 22:11:16.211842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.211448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" event={"ID":"e096a529-de3b-465f-8805-c0a68eaf0452","Type":"ContainerDied","Data":"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607"} Oct 29 22:11:16.211842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.211512 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78" event={"ID":"e096a529-de3b-465f-8805-c0a68eaf0452","Type":"ContainerDied","Data":"0d8f5d0ab2d9c05b831dfaed690bd2be7983890d4fa65e77ea7d2b761fa248ac"} Oct 29 22:11:16.211842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.211541 4543 scope.go:122] "RemoveContainer" containerID="ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607" Oct 29 22:11:16.213478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.213453 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" event={"ID":"facd82c7-d7d9-4e97-88bc-165887719d45","Type":"ContainerStarted","Data":"a97a5e93b03b3730676edbbb1b84665793fcb0fffae24178f6bd85ee0e6d2f3c"} Oct 29 22:11:16.213662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.213500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" event={"ID":"facd82c7-d7d9-4e97-88bc-165887719d45","Type":"ContainerStarted","Data":"b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef"} Oct 29 22:11:16.223403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.223386 4543 scope.go:122] "RemoveContainer" containerID="ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607" Oct 29 22:11:16.223684 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:11:16.223667 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607\": container with ID starting with ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607 not found: ID does not exist" containerID="ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607" Oct 29 22:11:16.223733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.223693 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607"} err="failed to get container status \"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607\": rpc error: code = NotFound desc = could not find container \"ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607\": container with ID starting with ead63fabe1ab9268347c55fa0d5d7ce4574e6fc8c196839971de0d47f5fdc607 not found: ID does not exist" Oct 29 22:11:16.272438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.272416 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 22:11:16.276980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.276961 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dnk78"] Oct 29 22:11:16.318229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.318188 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" podStartSLOduration=1.3181768489999999 podStartE2EDuration="1.318176849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:16.314430801 +0000 UTC m=+1738.056361471" watchObservedRunningTime="2026-10-29 22:11:16.318176849 +0000 UTC m=+1738.060107522" Oct 29 22:11:16.387799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:16.387767 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" path="/var/lib/kubelet/pods/e096a529-de3b-465f-8805-c0a68eaf0452/volumes" Oct 29 22:11:17.223658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:17.223624 4543 generic.go:356] "Generic (PLEG): container finished" podID="facd82c7-d7d9-4e97-88bc-165887719d45" containerID="a97a5e93b03b3730676edbbb1b84665793fcb0fffae24178f6bd85ee0e6d2f3c" exitCode=0 Oct 29 22:11:17.224108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:17.223710 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" event={"ID":"facd82c7-d7d9-4e97-88bc-165887719d45","Type":"ContainerDied","Data":"a97a5e93b03b3730676edbbb1b84665793fcb0fffae24178f6bd85ee0e6d2f3c"} Oct 29 22:11:18.434010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.433986 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:18.581406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.581305 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\" (UniqueName: \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\") pod \"facd82c7-d7d9-4e97-88bc-165887719d45\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " Oct 29 22:11:18.581601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.581548 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\") pod \"facd82c7-d7d9-4e97-88bc-165887719d45\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " Oct 29 22:11:18.581666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.581631 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\") pod \"facd82c7-d7d9-4e97-88bc-165887719d45\" (UID: \"facd82c7-d7d9-4e97-88bc-165887719d45\") " Oct 29 22:11:18.582537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.582151 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca" pod "facd82c7-d7d9-4e97-88bc-165887719d45" (UID: "facd82c7-d7d9-4e97-88bc-165887719d45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:18.582537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.582506 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:18.584049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.584023 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb" pod "facd82c7-d7d9-4e97-88bc-165887719d45" (UID: "facd82c7-d7d9-4e97-88bc-165887719d45"). InnerVolumeSpecName "kube-api-access-7dghb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:18.584734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.584709 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca" pod "facd82c7-d7d9-4e97-88bc-165887719d45" (UID: "facd82c7-d7d9-4e97-88bc-165887719d45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:18.684359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.684332 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/facd82c7-d7d9-4e97-88bc-165887719d45-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:18.684359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.684355 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dghb\" (UniqueName: \"kubernetes.io/projected/facd82c7-d7d9-4e97-88bc-165887719d45-kube-api-access-7dghb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:18.697848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.697819 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:11:18.697939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:18.697885 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:11:19.253625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:19.253577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" event={"ID":"facd82c7-d7d9-4e97-88bc-165887719d45","Type":"ContainerDied","Data":"b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef"} Oct 29 22:11:19.253835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:19.253681 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7b7e59f618b06b47b8c3094ff190a7fe410464aba7e18f7eaa4a558f3e771ef" Oct 29 22:11:19.253835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:19.253771 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8" Oct 29 22:11:21.287439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.287408 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:11:21.288124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288108 4543 cpu_manager.go:414] "containerMap: removing container" podUID="facd82c7-d7d9-4e97-88bc-165887719d45" containerName="revoke-keys" Oct 29 22:11:21.288124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288123 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facd82c7-d7d9-4e97-88bc-165887719d45" containerName="revoke-keys" Oct 29 22:11:21.288248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288146 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" Oct 29 22:11:21.288248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288152 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" Oct 29 22:11:21.288350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288284 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="facd82c7-d7d9-4e97-88bc-165887719d45" containerName="revoke-keys" Oct 29 22:11:21.288350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.288296 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096a529-de3b-465f-8805-c0a68eaf0452" containerName="istio-proxy" Oct 29 22:11:21.291416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.291390 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.310142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.310118 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05687d-openshift-default-dockercfg-l7t95\"" Oct 29 22:11:21.329573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.329548 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:11:21.363533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363495 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.363674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363544 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.363763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363687 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.363825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363761 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.363825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363810 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.363987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.363964 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tsj4\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.364129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.364110 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.364211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.364167 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.364279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.364252 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.465762 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.465852 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.465933 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466056 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466121 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466161 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466200 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466261 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.467093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.466402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsj4\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.473100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.467855 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.473100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.471835 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.473100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.472128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.473100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.472400 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.473100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.472648 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.477240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.477191 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.492584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.492550 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.544744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.538629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.544744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.543715 4543 scope.go:122] "RemoveContainer" containerID="5bd8a2cc316faeca4fbbb55f07492a9417251e5da3ea8af2993ecdfe736cb002" Oct 29 22:11:21.550157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.549931 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsj4\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.569821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.569797 4543 scope.go:122] "RemoveContainer" containerID="b0b02a7acbe99ec81cf13819d078932077c8d11cdb55c04c8e0a6627bc42c004" Oct 29 22:11:21.584038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.584011 4543 scope.go:122] "RemoveContainer" containerID="43b6518c401112baed1eb983116ddab0c1b3bd1cdd2b57be8aaecd28b54087d7" Oct 29 22:11:21.615319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.615286 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:21.973428 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:21.973398 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c08613e_80d1_423f_a3d3_4a5532c6dc54.slice/crio-aacb1bf7c132a1515bc4c10ff1914725e84b99a2af56a717fd98f9adfc467abf WatchSource:0}: Error finding container aacb1bf7c132a1515bc4c10ff1914725e84b99a2af56a717fd98f9adfc467abf: Status 404 returned error can't find the container with id aacb1bf7c132a1515bc4c10ff1914725e84b99a2af56a717fd98f9adfc467abf Oct 29 22:11:21.974858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.974827 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:11:21.987966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.987928 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:21.988066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.987997 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:21.988066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:21.988025 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:11:22.284858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:22.284768 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerStarted","Data":"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932"} Oct 29 22:11:22.284858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:22.284830 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerStarted","Data":"aacb1bf7c132a1515bc4c10ff1914725e84b99a2af56a717fd98f9adfc467abf"} Oct 29 22:11:22.338147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:22.338093 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podStartSLOduration=1.338061583 podStartE2EDuration="1.338061583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:22.337439584 +0000 UTC m=+1744.079370257" watchObservedRunningTime="2026-10-29 22:11:22.338061583 +0000 UTC m=+1744.079992255" Oct 29 22:11:22.623415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:22.621212 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:23.622689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:23.622658 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:24.306089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:24.305599 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:24.306602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:24.306578 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:11:26.740680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:26.740632 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded" Oct 29 22:11:26.753352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:26.753305 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:26.833118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:26.833050 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:26.988210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:26.988162 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.123152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.122180 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.305101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.282858 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.384198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.307168 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:11:27.485431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.445443 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Oct 29 22:11:27.572476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.489170 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.572935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.489830 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:11:27.573275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.571606 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 29 22:11:27.757260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.757219 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.771739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.771706 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:11:27.950150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.862652 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.950150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.947065 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:11:27.950150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:27.862689 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": context deadline exceeded" Oct 29 22:11:28.017156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.003156 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:11:28.017156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.003591 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:11:28.178388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.178341 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:11:28.228206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.228162 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:11:28.295155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.291156 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 29 22:11:28.393341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.393297 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:11:28.502611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.502566 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.582915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.582833 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.600094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.583193 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:11:28.600664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.600633 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639539 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639608 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639646 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639680 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639713 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:11:28.646194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.639743 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:11:28.756700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.754162 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:11:28.765514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.765488 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:11:28.772561 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:11:28.772531 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:11:28.786290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.786237 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:11:28.787745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.787717 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:11:28.790489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.790472 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:11:28.810135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.804747 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:11:28.810135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.809776 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:11:28.816352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:28.810814 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:11:30.359533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.359328 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26"] Oct 29 22:11:30.366174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.366149 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.372560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.372525 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26"] Oct 29 22:11:30.387110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.387044 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:30.387293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.387128 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:30.412278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.409544 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:30.412278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.409681 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:30.454555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.454498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.454761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.454739 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925bh\" (UniqueName: \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.454844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.454780 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.556313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.556269 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.561178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.561144 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.561412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.561203 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925bh\" (UniqueName: \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.561613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.561593 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.564919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.564889 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.594649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.594618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-925bh\" (UniqueName: \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\") pod \"maas-api-revoke-sub-9977774f-n9l26\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:30.731018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:30.730980 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:31.062700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:31.062499 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26"] Oct 29 22:11:31.062700 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:31.062543 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95880d21_2ff4_44e8_a166_a23e769bea44.slice/crio-ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c WatchSource:0}: Error finding container ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c: Status 404 returned error can't find the container with id ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c Oct 29 22:11:31.857814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:31.857776 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" event={"ID":"95880d21-2ff4-44e8-a166-a23e769bea44","Type":"ContainerStarted","Data":"1a824072c231b773f35e6b3215d38ba4fdbeac57d005edccabc7069368506a62"} Oct 29 22:11:31.858337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:31.857841 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" event={"ID":"95880d21-2ff4-44e8-a166-a23e769bea44","Type":"ContainerStarted","Data":"ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c"} Oct 29 22:11:31.902801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:31.902749 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" podStartSLOduration=1.902735203 podStartE2EDuration="1.902735203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:31.900325796 +0000 UTC m=+1753.642256492" watchObservedRunningTime="2026-10-29 22:11:31.902735203 +0000 UTC m=+1753.644665876" Oct 29 22:11:32.868026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:32.867986 4543 generic.go:356] "Generic (PLEG): container finished" podID="95880d21-2ff4-44e8-a166-a23e769bea44" containerID="1a824072c231b773f35e6b3215d38ba4fdbeac57d005edccabc7069368506a62" exitCode=0 Oct 29 22:11:32.868026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:32.868016 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" event={"ID":"95880d21-2ff4-44e8-a166-a23e769bea44","Type":"ContainerDied","Data":"1a824072c231b773f35e6b3215d38ba4fdbeac57d005edccabc7069368506a62"} Oct 29 22:11:34.038386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.038360 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:34.114841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.114814 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\") pod \"95880d21-2ff4-44e8-a166-a23e769bea44\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " Oct 29 22:11:34.115004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.114945 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\") pod \"95880d21-2ff4-44e8-a166-a23e769bea44\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " Oct 29 22:11:34.115004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.114997 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\" (UniqueName: \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\") pod \"95880d21-2ff4-44e8-a166-a23e769bea44\" (UID: \"95880d21-2ff4-44e8-a166-a23e769bea44\") " Oct 29 22:11:34.115407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.115378 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca" pod "95880d21-2ff4-44e8-a166-a23e769bea44" (UID: "95880d21-2ff4-44e8-a166-a23e769bea44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:34.115652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.115623 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:34.117455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.117426 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh" pod "95880d21-2ff4-44e8-a166-a23e769bea44" (UID: "95880d21-2ff4-44e8-a166-a23e769bea44"). InnerVolumeSpecName "kube-api-access-925bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:34.117656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.117622 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca" pod "95880d21-2ff4-44e8-a166-a23e769bea44" (UID: "95880d21-2ff4-44e8-a166-a23e769bea44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:34.217382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.217357 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-925bh\" (UniqueName: \"kubernetes.io/projected/95880d21-2ff4-44e8-a166-a23e769bea44-kube-api-access-925bh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:34.217382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.217382 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95880d21-2ff4-44e8-a166-a23e769bea44-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:34.887474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.887447 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" Oct 29 22:11:34.887662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.887475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26" event={"ID":"95880d21-2ff4-44e8-a166-a23e769bea44","Type":"ContainerDied","Data":"ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c"} Oct 29 22:11:34.887662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:34.887520 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba74d43c748e886cf50ff8a24ad298c21a1c3ac47a044b5c4fd6511406ec500c" Oct 29 22:11:36.165594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.165571 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:11:36.169234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.169215 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:11:36.239635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.239604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\" (UniqueName: \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\") pod \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " Oct 29 22:11:36.239814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.239697 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\") pod \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\" (UID: \"ae8247e7-df02-4f5c-98e3-cc6343bc383d\") " Oct 29 22:11:36.239888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.239822 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\") pod \"3ef5ad51-dddd-4d46-b244-44da697c900c\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " Oct 29 22:11:36.239888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.239854 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\" (UniqueName: \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\") pod \"3ef5ad51-dddd-4d46-b244-44da697c900c\" (UID: \"3ef5ad51-dddd-4d46-b244-44da697c900c\") " Oct 29 22:11:36.240277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.240250 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume" pod "ae8247e7-df02-4f5c-98e3-cc6343bc383d" (UID: "ae8247e7-df02-4f5c-98e3-cc6343bc383d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:36.240277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.240249 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume" pod "3ef5ad51-dddd-4d46-b244-44da697c900c" (UID: "3ef5ad51-dddd-4d46-b244-44da697c900c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:36.240656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.240634 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8247e7-df02-4f5c-98e3-cc6343bc383d-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:36.240782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.240660 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5ad51-dddd-4d46-b244-44da697c900c-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:36.242331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.242308 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6" pod "ae8247e7-df02-4f5c-98e3-cc6343bc383d" (UID: "ae8247e7-df02-4f5c-98e3-cc6343bc383d"). InnerVolumeSpecName "kube-api-access-hw6z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:36.242503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.242486 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk" pod "3ef5ad51-dddd-4d46-b244-44da697c900c" (UID: "3ef5ad51-dddd-4d46-b244-44da697c900c"). InnerVolumeSpecName "kube-api-access-cwbmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:36.296771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.296739 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29"] Oct 29 22:11:36.297889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.297862 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" Oct 29 22:11:36.297889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.297889 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" Oct 29 22:11:36.298114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.297956 4543 cpu_manager.go:414] "containerMap: removing container" podUID="95880d21-2ff4-44e8-a166-a23e769bea44" containerName="revoke-keys" Oct 29 22:11:36.298114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.297964 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95880d21-2ff4-44e8-a166-a23e769bea44" containerName="revoke-keys" Oct 29 22:11:36.298114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.297994 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" Oct 29 22:11:36.298114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.298003 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" Oct 29 22:11:36.298352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.298212 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="95880d21-2ff4-44e8-a166-a23e769bea44" containerName="revoke-keys" Oct 29 22:11:36.298352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.298230 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerName="payload-pre-processing" Oct 29 22:11:36.298352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.298245 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerName="payload-processing" Oct 29 22:11:36.302697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.302672 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.320311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.320284 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:36.320436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.320338 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:36.320486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.320467 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:36.320525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.320486 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:36.321649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.321632 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29"] Oct 29 22:11:36.342058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.342033 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.342199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.342160 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h66v\" (UniqueName: \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.342199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.342187 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.342326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.342317 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbmk\" (UniqueName: \"kubernetes.io/projected/3ef5ad51-dddd-4d46-b244-44da697c900c-kube-api-access-cwbmk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:36.342392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.342329 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw6z6\" (UniqueName: \"kubernetes.io/projected/ae8247e7-df02-4f5c-98e3-cc6343bc383d-kube-api-access-hw6z6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:36.443742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.443717 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.443835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.443797 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h66v\" (UniqueName: \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.443835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.443823 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.444347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.444325 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.446354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.446335 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.469568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.469541 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h66v\" (UniqueName: \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\") pod \"maas-api-revoke-sub-19d58095-dfd29\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.624415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.624390 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:36.904742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.904712 4543 generic.go:356] "Generic (PLEG): container finished" podID="3ef5ad51-dddd-4d46-b244-44da697c900c" containerID="33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6" exitCode=0 Oct 29 22:11:36.904900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.904837 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" Oct 29 22:11:36.904900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.904857 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" event={"ID":"3ef5ad51-dddd-4d46-b244-44da697c900c","Type":"ContainerDied","Data":"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6"} Oct 29 22:11:36.905019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.904925 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9" event={"ID":"3ef5ad51-dddd-4d46-b244-44da697c900c","Type":"ContainerDied","Data":"bc660e6d8cc036bb7b4b5c205d4adeca10ac364568ebf127a86135b927771dd8"} Oct 29 22:11:36.905019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.904967 4543 scope.go:122] "RemoveContainer" containerID="33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6" Oct 29 22:11:36.906448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.906419 4543 generic.go:356] "Generic (PLEG): container finished" podID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" containerID="f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195" exitCode=0 Oct 29 22:11:36.906582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.906483 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" Oct 29 22:11:36.906582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.906483 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" event={"ID":"ae8247e7-df02-4f5c-98e3-cc6343bc383d","Type":"ContainerDied","Data":"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195"} Oct 29 22:11:36.906582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.906542 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j" event={"ID":"ae8247e7-df02-4f5c-98e3-cc6343bc383d","Type":"ContainerDied","Data":"85be86e83bb1b508f3a62ab01bf54e55f0cdbf239f5727b899ab8c18905dc8a3"} Oct 29 22:11:36.916293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.916274 4543 scope.go:122] "RemoveContainer" containerID="33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6" Oct 29 22:11:36.916547 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:11:36.916529 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6\": container with ID starting with 33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6 not found: ID does not exist" containerID="33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6" Oct 29 22:11:36.916617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.916557 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6"} err="failed to get container status \"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6\": rpc error: code = NotFound desc = could not find container \"33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6\": container with ID starting with 33c3861edd81fc801fe560447bcd56a46ed5c127ea61349e8250a690a9ffd2f6 not found: ID does not exist" Oct 29 22:11:36.916617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.916574 4543 scope.go:122] "RemoveContainer" containerID="f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195" Oct 29 22:11:36.926665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.926647 4543 scope.go:122] "RemoveContainer" containerID="f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195" Oct 29 22:11:36.926947 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:11:36.926929 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195\": container with ID starting with f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195 not found: ID does not exist" containerID="f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195" Oct 29 22:11:36.927014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.926953 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195"} err="failed to get container status \"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195\": rpc error: code = NotFound desc = could not find container \"f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195\": container with ID starting with f036160736a0ca97aea73c043a082199c1a8d462fad529c63b4ebebeb8cbb195 not found: ID does not exist" Oct 29 22:11:36.965032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.965007 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:11:36.974449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.974427 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29"] Oct 29 22:11:36.978110 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:36.978066 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dc3dbd_c2f3_40d8_8a86_a98d81dc0655.slice/crio-067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7 WatchSource:0}: Error finding container 067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7: Status 404 returned error can't find the container with id 067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7 Oct 29 22:11:36.979550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:36.979530 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-796d71-d8996c956-dtvg9"] Oct 29 22:11:37.011475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:37.011453 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:11:37.021529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:37.021509 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-796d71-6bd764db55-w7m5j"] Oct 29 22:11:37.917080 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:37.917049 4543 generic.go:356] "Generic (PLEG): container finished" podID="c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" containerID="2748f6945d9592c74bc9039787c3517b603ca71fd81e9632d6105bcecde71e90" exitCode=0 Oct 29 22:11:37.917444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:37.917134 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" event={"ID":"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655","Type":"ContainerDied","Data":"2748f6945d9592c74bc9039787c3517b603ca71fd81e9632d6105bcecde71e90"} Oct 29 22:11:37.917444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:37.917183 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" event={"ID":"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655","Type":"ContainerStarted","Data":"067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7"} Oct 29 22:11:38.419454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:38.419421 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef5ad51-dddd-4d46-b244-44da697c900c" path="/var/lib/kubelet/pods/3ef5ad51-dddd-4d46-b244-44da697c900c/volumes" Oct 29 22:11:38.419913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:38.419897 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8247e7-df02-4f5c-98e3-cc6343bc383d" path="/var/lib/kubelet/pods/ae8247e7-df02-4f5c-98e3-cc6343bc383d/volumes" Oct 29 22:11:39.105561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.105533 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:39.182024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.181962 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\" (UniqueName: \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\") pod \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " Oct 29 22:11:39.182167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.182117 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\") pod \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " Oct 29 22:11:39.182167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.182154 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\") pod \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\" (UID: \"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655\") " Oct 29 22:11:39.182607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.182569 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca" pod "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" (UID: "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:39.184446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.184418 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v" pod "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" (UID: "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655"). InnerVolumeSpecName "kube-api-access-4h66v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:39.184446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.184437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca" pod "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" (UID: "c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:39.283907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.283879 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h66v\" (UniqueName: \"kubernetes.io/projected/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-kube-api-access-4h66v\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:39.283907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.283902 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:39.284145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.283912 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:39.936612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.936584 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" Oct 29 22:11:39.936787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.936610 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29" event={"ID":"c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655","Type":"ContainerDied","Data":"067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7"} Oct 29 22:11:39.936787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:39.936659 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="067ff4a574128c076129a326e74657e11649832e890b1b8e2290afb9ed6c75b7" Oct 29 22:11:47.761116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.761061 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:11:47.762039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.762013 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" containerName="revoke-keys" Oct 29 22:11:47.762039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.762040 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" containerName="revoke-keys" Oct 29 22:11:47.762284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.762255 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" containerName="revoke-keys" Oct 29 22:11:47.765315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.765286 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:47.784987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.784961 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:11:47.791693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.791672 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-99e420\"" Oct 29 22:11:47.791865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.791751 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-99e420-dockercfg-r9r6g\"" Oct 29 22:11:47.977098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.972954 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:47.977098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.973138 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqw46\" (UniqueName: \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:47.977098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.975052 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:11:47.992633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.992606 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:47.999498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:47.999468 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:11:48.074335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.074244 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqw46\" (UniqueName: \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:48.074335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.074309 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.074628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.074426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:48.074628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.074446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gds8t\" (UniqueName: \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.075502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.075476 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:48.100673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.100651 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqw46\" (UniqueName: \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\") pod \"payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:48.175353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.175318 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.175538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.175517 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gds8t\" (UniqueName: \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.176133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.176065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.201302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.201275 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gds8t\" (UniqueName: \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\") pod \"payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.346394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.346327 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:48.381203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.381175 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:48.675951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.675921 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:11:48.676114 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:48.676041 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864b9a10_f831_472b_bd09_390d86d11f6a.slice/crio-d36c0dc25bd01a1e9ed45103a2c7e94a51762bd20e92883ac27b10293886ec76 WatchSource:0}: Error finding container d36c0dc25bd01a1e9ed45103a2c7e94a51762bd20e92883ac27b10293886ec76: Status 404 returned error can't find the container with id d36c0dc25bd01a1e9ed45103a2c7e94a51762bd20e92883ac27b10293886ec76 Oct 29 22:11:48.733809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:48.733784 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:11:49.048060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.048024 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" event={"ID":"03fac3a7-d247-4585-b83f-4bae00efdb4b","Type":"ContainerStarted","Data":"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7"} Oct 29 22:11:49.048542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.048098 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" event={"ID":"03fac3a7-d247-4585-b83f-4bae00efdb4b","Type":"ContainerStarted","Data":"54ed2a9177c6168d4b446168c38e189608b0acb4c8d0b6c90d7a5e1c96c6730c"} Oct 29 22:11:49.048542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.048132 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:11:49.050192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.050158 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" event={"ID":"864b9a10-f831-472b-bd09-390d86d11f6a","Type":"ContainerStarted","Data":"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a"} Oct 29 22:11:49.050317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.050208 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" event={"ID":"864b9a10-f831-472b-bd09-390d86d11f6a","Type":"ContainerStarted","Data":"d36c0dc25bd01a1e9ed45103a2c7e94a51762bd20e92883ac27b10293886ec76"} Oct 29 22:11:49.050317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.050229 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:11:49.092868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.092811 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" podStartSLOduration=2.092798206 podStartE2EDuration="2.092798206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:49.091626579 +0000 UTC m=+1770.833557251" watchObservedRunningTime="2026-10-29 22:11:49.092798206 +0000 UTC m=+1770.834728878" Oct 29 22:11:49.139018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:49.138972 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" podStartSLOduration=2.138960083 podStartE2EDuration="2.138960083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:49.135194039 +0000 UTC m=+1770.877124737" watchObservedRunningTime="2026-10-29 22:11:49.138960083 +0000 UTC m=+1770.880890790" Oct 29 22:11:53.958788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.958754 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns"] Oct 29 22:11:53.963344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.963321 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:53.981434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.981411 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns"] Oct 29 22:11:53.989194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.989169 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:11:53.989519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.989488 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:11:53.989644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.989557 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:11:53.993224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:53.993205 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:11:54.066805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.066772 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.066927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.066903 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.066979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.066953 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp8h\" (UniqueName: \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.168740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.168713 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.168883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.168793 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.168883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.168833 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp8h\" (UniqueName: \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.169343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.169321 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.171278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.171256 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.201688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.201654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp8h\" (UniqueName: \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\") pod \"maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.279739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.279696 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:54.615856 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:11:54.615825 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bec62e_2d56_487d_938b_ee6deee44bba.slice/crio-5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005 WatchSource:0}: Error finding container 5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005: Status 404 returned error can't find the container with id 5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005 Oct 29 22:11:54.615856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:54.615830 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns"] Oct 29 22:11:55.102038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:55.102005 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" event={"ID":"98bec62e-2d56-487d-938b-ee6deee44bba","Type":"ContainerStarted","Data":"7a79610016ccad20ca01dc8289c85efe00badd72330d4bedf17c99d8453982f5"} Oct 29 22:11:55.102408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:55.102070 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" event={"ID":"98bec62e-2d56-487d-938b-ee6deee44bba","Type":"ContainerStarted","Data":"5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005"} Oct 29 22:11:55.152509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:55.152451 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" podStartSLOduration=2.152436058 podStartE2EDuration="2.152436058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:55.149416335 +0000 UTC m=+1776.891347043" watchObservedRunningTime="2026-10-29 22:11:55.152436058 +0000 UTC m=+1776.894366754" Oct 29 22:11:56.111906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:56.111865 4543 generic.go:356] "Generic (PLEG): container finished" podID="98bec62e-2d56-487d-938b-ee6deee44bba" containerID="7a79610016ccad20ca01dc8289c85efe00badd72330d4bedf17c99d8453982f5" exitCode=0 Oct 29 22:11:56.112357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:56.111946 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" event={"ID":"98bec62e-2d56-487d-938b-ee6deee44bba","Type":"ContainerDied","Data":"7a79610016ccad20ca01dc8289c85efe00badd72330d4bedf17c99d8453982f5"} Oct 29 22:11:56.393272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:56.393187 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:11:56.393464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:56.393421 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" containerID="cri-o://30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b" gracePeriod=30 Oct 29 22:11:57.299591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.299563 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:57.315983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.315954 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\") pod \"98bec62e-2d56-487d-938b-ee6deee44bba\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " Oct 29 22:11:57.316182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.315993 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\") pod \"98bec62e-2d56-487d-938b-ee6deee44bba\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " Oct 29 22:11:57.316182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.316021 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\" (UniqueName: \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\") pod \"98bec62e-2d56-487d-938b-ee6deee44bba\" (UID: \"98bec62e-2d56-487d-938b-ee6deee44bba\") " Oct 29 22:11:57.316453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.316425 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca" pod "98bec62e-2d56-487d-938b-ee6deee44bba" (UID: "98bec62e-2d56-487d-938b-ee6deee44bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:57.316728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.316707 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:57.318370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.318342 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca" pod "98bec62e-2d56-487d-938b-ee6deee44bba" (UID: "98bec62e-2d56-487d-938b-ee6deee44bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:57.318370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.318359 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h" pod "98bec62e-2d56-487d-938b-ee6deee44bba" (UID: "98bec62e-2d56-487d-938b-ee6deee44bba"). InnerVolumeSpecName "kube-api-access-ltp8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:57.417947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.417865 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltp8h\" (UniqueName: \"kubernetes.io/projected/98bec62e-2d56-487d-938b-ee6deee44bba-kube-api-access-ltp8h\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:57.417947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:57.417896 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bec62e-2d56-487d-938b-ee6deee44bba-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:11:57.961190 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:11:57.961159 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:11:58.132642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:58.132617 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" Oct 29 22:11:58.132834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:58.132652 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns" event={"ID":"98bec62e-2d56-487d-938b-ee6deee44bba","Type":"ContainerDied","Data":"5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005"} Oct 29 22:11:58.132834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:11:58.132695 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5086f3b11b03cb1ba529493050fcaec7be5aac86dd01d00e83afad4ce6e69005" Oct 29 22:12:00.062932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:00.062903 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:12:00.063552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:00.063518 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:12:01.680701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.680677 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:12:01.785629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785585 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.785847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785682 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.785847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785725 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.785847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785759 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.786206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.786206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.785928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.786206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.786007 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.786206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.786129 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.786206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.786174 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\") pod \"105252e5-1578-4832-91c0-0a1e8303b177\" (UID: \"105252e5-1578-4832-91c0-0a1e8303b177\") " Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.857428 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.859432 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.859695 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.860435 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "kube-api-access-h6vmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.860664 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.872195 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.902891 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.906974 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:01.928234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.912239 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo" pod "105252e5-1578-4832-91c0-0a1e8303b177" (UID: "105252e5-1578-4832-91c0-0a1e8303b177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938611 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vmz\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-kube-api-access-h6vmz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938652 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938667 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938681 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105252e5-1578-4832-91c0-0a1e8303b177-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938693 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105252e5-1578-4832-91c0-0a1e8303b177-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938705 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938717 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105252e5-1578-4832-91c0-0a1e8303b177-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938729 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.938744 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105252e5-1578-4832-91c0-0a1e8303b177-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.970586 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52"] Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.971736 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98bec62e-2d56-487d-938b-ee6deee44bba" containerName="revoke-keys" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.971761 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bec62e-2d56-487d-938b-ee6deee44bba" containerName="revoke-keys" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.971799 4543 cpu_manager.go:414] "containerMap: removing container" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.971806 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.972055 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bec62e-2d56-487d-938b-ee6deee44bba" containerName="revoke-keys" Oct 29 22:12:01.976892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.972102 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="105252e5-1578-4832-91c0-0a1e8303b177" containerName="istio-proxy" Oct 29 22:12:01.977959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.977104 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:01.980946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.980237 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52"] Oct 29 22:12:01.997770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.997144 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:12:01.997924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.997878 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:12:01.997997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.997947 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:12:01.998057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:01.998013 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:12:02.040108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.040048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpsqs\" (UniqueName: \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.040280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.040122 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.040324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.040269 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.142047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.142008 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.142275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.142242 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpsqs\" (UniqueName: \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.142354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.142299 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.142938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.142890 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.144534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.144512 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.169261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.169232 4543 generic.go:356] "Generic (PLEG): container finished" podID="105252e5-1578-4832-91c0-0a1e8303b177" containerID="30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b" exitCode=0 Oct 29 22:12:02.169372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.169296 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" event={"ID":"105252e5-1578-4832-91c0-0a1e8303b177","Type":"ContainerDied","Data":"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b"} Oct 29 22:12:02.169372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.169355 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" event={"ID":"105252e5-1578-4832-91c0-0a1e8303b177","Type":"ContainerDied","Data":"273abcca580824db210c8421d182b7e1895e96b2f63729c8cc3bf48f3ce83d40"} Oct 29 22:12:02.169372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.169362 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq" Oct 29 22:12:02.169549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.169378 4543 scope.go:122] "RemoveContainer" containerID="30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b" Oct 29 22:12:02.170505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.170486 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpsqs\" (UniqueName: \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\") pod \"maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.181205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.181184 4543 scope.go:122] "RemoveContainer" containerID="30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b" Oct 29 22:12:02.181473 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:02.181456 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b\": container with ID starting with 30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b not found: ID does not exist" containerID="30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b" Oct 29 22:12:02.181531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.181489 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b"} err="failed to get container status \"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b\": rpc error: code = NotFound desc = could not find container \"30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b\": container with ID starting with 30726d1943a34e6dc69b70982806fe0ac45910ae30a4c69e1f601900df98b30b not found: ID does not exist" Oct 29 22:12:02.233673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.233650 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:12:02.239791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.239772 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fe45b-openshift-default-7986688dbf-npzgq"] Oct 29 22:12:02.305653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.305631 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:02.391741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.391711 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105252e5-1578-4832-91c0-0a1e8303b177" path="/var/lib/kubelet/pods/105252e5-1578-4832-91c0-0a1e8303b177/volumes" Oct 29 22:12:02.615187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:02.615158 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52"] Oct 29 22:12:02.615293 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:02.615204 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9897a756_8532_4d01_98cf_91df5fafb4b2.slice/crio-cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8 WatchSource:0}: Error finding container cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8: Status 404 returned error can't find the container with id cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8 Oct 29 22:12:03.191983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:03.191884 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" event={"ID":"9897a756-8532-4d01-98cf-91df5fafb4b2","Type":"ContainerStarted","Data":"e0eb44e6b228b10465a97607658713de30ec68acdf2cf347b2ad2e7ad9634040"} Oct 29 22:12:03.192613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:03.192585 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" event={"ID":"9897a756-8532-4d01-98cf-91df5fafb4b2","Type":"ContainerStarted","Data":"cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8"} Oct 29 22:12:03.277189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:03.277123 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" podStartSLOduration=2.277102488 podStartE2EDuration="2.277102488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:03.247791046 +0000 UTC m=+1784.989721727" watchObservedRunningTime="2026-10-29 22:12:03.277102488 +0000 UTC m=+1785.019033162" Oct 29 22:12:04.215510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:04.215455 4543 generic.go:356] "Generic (PLEG): container finished" podID="9897a756-8532-4d01-98cf-91df5fafb4b2" containerID="e0eb44e6b228b10465a97607658713de30ec68acdf2cf347b2ad2e7ad9634040" exitCode=0 Oct 29 22:12:04.216112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:04.215552 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" event={"ID":"9897a756-8532-4d01-98cf-91df5fafb4b2","Type":"ContainerDied","Data":"e0eb44e6b228b10465a97607658713de30ec68acdf2cf347b2ad2e7ad9634040"} Oct 29 22:12:04.785100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:04.778297 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:12:04.834194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:04.829730 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:12:05.458321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.458297 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:05.538064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.537983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\" (UniqueName: \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\") pod \"9897a756-8532-4d01-98cf-91df5fafb4b2\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " Oct 29 22:12:05.538064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.538041 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\") pod \"9897a756-8532-4d01-98cf-91df5fafb4b2\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " Oct 29 22:12:05.538377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.538327 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\") pod \"9897a756-8532-4d01-98cf-91df5fafb4b2\" (UID: \"9897a756-8532-4d01-98cf-91df5fafb4b2\") " Oct 29 22:12:05.538767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.538737 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca" pod "9897a756-8532-4d01-98cf-91df5fafb4b2" (UID: "9897a756-8532-4d01-98cf-91df5fafb4b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:05.539641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.539611 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:05.540605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.540574 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca" pod "9897a756-8532-4d01-98cf-91df5fafb4b2" (UID: "9897a756-8532-4d01-98cf-91df5fafb4b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:05.540720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.540625 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs" pod "9897a756-8532-4d01-98cf-91df5fafb4b2" (UID: "9897a756-8532-4d01-98cf-91df5fafb4b2"). InnerVolumeSpecName "kube-api-access-gpsqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:05.643387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.643342 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpsqs\" (UniqueName: \"kubernetes.io/projected/9897a756-8532-4d01-98cf-91df5fafb4b2-kube-api-access-gpsqs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:05.643387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:05.643383 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9897a756-8532-4d01-98cf-91df5fafb4b2-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:06.260925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:06.260896 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" Oct 29 22:12:06.261106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:06.260935 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52" event={"ID":"9897a756-8532-4d01-98cf-91df5fafb4b2","Type":"ContainerDied","Data":"cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8"} Oct 29 22:12:06.261106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:06.261010 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6500fa758f587667b3f32f7e640b5b6d703cf8369bd1f9f0f1d61d146690f8" Oct 29 22:12:09.283687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:09.283652 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns"] Oct 29 22:12:09.294948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:09.294922 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80bd36-qn2ns"] Oct 29 22:12:09.780017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:09.779947 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" podUID="b98860da-1de2-489d-aeaf-e1968915a001" containerName="payload-processing" containerID="cri-o://1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117" gracePeriod=25 Oct 29 22:12:09.830684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:09.830648 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" podUID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" containerName="payload-pre-processing" containerID="cri-o://36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190" gracePeriod=25 Oct 29 22:12:10.082194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.080800 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k"] Oct 29 22:12:10.114762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.114428 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-796d71-42l6k"] Oct 29 22:12:10.172898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.172866 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:12:10.176675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.176653 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:12:10.243470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.243432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\") pod \"b98860da-1de2-489d-aeaf-e1968915a001\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " Oct 29 22:12:10.243734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.243538 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\" (UniqueName: \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\") pod \"b98860da-1de2-489d-aeaf-e1968915a001\" (UID: \"b98860da-1de2-489d-aeaf-e1968915a001\") " Oct 29 22:12:10.243734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.243626 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\") pod \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " Oct 29 22:12:10.243734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.243665 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\" (UniqueName: \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\") pod \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\" (UID: \"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7\") " Oct 29 22:12:10.244701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.244670 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume" pod "33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" (UID: "33be8328-cdbb-4f7f-8d6e-4ec08bce7af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:10.244882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.244860 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume" pod "b98860da-1de2-489d-aeaf-e1968915a001" (UID: "b98860da-1de2-489d-aeaf-e1968915a001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:10.245464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.245438 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98860da-1de2-489d-aeaf-e1968915a001-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.245545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.245470 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.246893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.246857 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm" pod "b98860da-1de2-489d-aeaf-e1968915a001" (UID: "b98860da-1de2-489d-aeaf-e1968915a001"). InnerVolumeSpecName "kube-api-access-vjfvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:10.247584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.247554 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9" pod "33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" (UID: "33be8328-cdbb-4f7f-8d6e-4ec08bce7af7"). InnerVolumeSpecName "kube-api-access-dxvs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:10.302180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.302152 4543 generic.go:356] "Generic (PLEG): container finished" podID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" containerID="36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190" exitCode=0 Oct 29 22:12:10.302581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.302231 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" Oct 29 22:12:10.302581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.302235 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" event={"ID":"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7","Type":"ContainerDied","Data":"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190"} Oct 29 22:12:10.302581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.302317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54" event={"ID":"33be8328-cdbb-4f7f-8d6e-4ec08bce7af7","Type":"ContainerDied","Data":"ea0dce03ace66a6fee858abe72781f567b03f30c6fa5b663ea707149010b5963"} Oct 29 22:12:10.302581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.302342 4543 scope.go:122] "RemoveContainer" containerID="36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190" Oct 29 22:12:10.304401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.304316 4543 generic.go:356] "Generic (PLEG): container finished" podID="b98860da-1de2-489d-aeaf-e1968915a001" containerID="1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117" exitCode=0 Oct 29 22:12:10.304401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.304401 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" event={"ID":"b98860da-1de2-489d-aeaf-e1968915a001","Type":"ContainerDied","Data":"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117"} Oct 29 22:12:10.304401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.304400 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" Oct 29 22:12:10.304657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.304440 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4" event={"ID":"b98860da-1de2-489d-aeaf-e1968915a001","Type":"ContainerDied","Data":"d39207c6a5f1891e80df7ada63d81b939453f283d251345994a7f4e3e3215076"} Oct 29 22:12:10.314398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.314378 4543 scope.go:122] "RemoveContainer" containerID="36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190" Oct 29 22:12:10.314655 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:10.314636 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190\": container with ID starting with 36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190 not found: ID does not exist" containerID="36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190" Oct 29 22:12:10.314755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.314661 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190"} err="failed to get container status \"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190\": rpc error: code = NotFound desc = could not find container \"36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190\": container with ID starting with 36c87edf86a4f12ddcbf12daf544d216ea79dc86a5f6932829df1116f11c8190 not found: ID does not exist" Oct 29 22:12:10.314755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.314679 4543 scope.go:122] "RemoveContainer" containerID="1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117" Oct 29 22:12:10.328737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.328699 4543 scope.go:122] "RemoveContainer" containerID="1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117" Oct 29 22:12:10.329050 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:10.329030 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117\": container with ID starting with 1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117 not found: ID does not exist" containerID="1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117" Oct 29 22:12:10.329177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.329059 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117"} err="failed to get container status \"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117\": rpc error: code = NotFound desc = could not find container \"1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117\": container with ID starting with 1dcd3ff5c256bfa3f5926074deb764f6305e06a8ebca90cf0af68cdd5bd01117 not found: ID does not exist" Oct 29 22:12:10.346749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.346702 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvs9\" (UniqueName: \"kubernetes.io/projected/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7-kube-api-access-dxvs9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.346749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.346723 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfvm\" (UniqueName: \"kubernetes.io/projected/b98860da-1de2-489d-aeaf-e1968915a001-kube-api-access-vjfvm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.365792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.365711 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:12:10.370578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.370547 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-80bd36-5d5445f79d-nthb4"] Oct 29 22:12:10.387048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.387024 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db" path="/var/lib/kubelet/pods/4ba591fe-cc1d-4c42-8aa1-2c9e77dab5db/volumes" Oct 29 22:12:10.387524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.387508 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bec62e-2d56-487d-938b-ee6deee44bba" path="/var/lib/kubelet/pods/98bec62e-2d56-487d-938b-ee6deee44bba/volumes" Oct 29 22:12:10.387919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.387906 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98860da-1de2-489d-aeaf-e1968915a001" path="/var/lib/kubelet/pods/b98860da-1de2-489d-aeaf-e1968915a001/volumes" Oct 29 22:12:10.407771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.407753 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:12:10.413432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:10.413413 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-80bd36-694bf5995-zvq54"] Oct 29 22:12:12.386497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:12.386462 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" path="/var/lib/kubelet/pods/33be8328-cdbb-4f7f-8d6e-4ec08bce7af7/volumes" Oct 29 22:12:15.285509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.285474 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:12:15.285954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.285713 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" podUID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" containerName="payload-processing" containerID="cri-o://dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6" gracePeriod=30 Oct 29 22:12:15.311562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.311532 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:12:15.311729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.311708 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" podUID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" containerName="payload-pre-processing" containerID="cri-o://8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f" gracePeriod=30 Oct 29 22:12:15.699912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.699662 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:12:15.701612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.701588 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:12:15.762945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.762912 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\") pod \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " Oct 29 22:12:15.763278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.763253 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\") pod \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " Oct 29 22:12:15.763431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.763417 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\" (UniqueName: \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\") pod \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\" (UID: \"0ed3194e-7db5-40f4-8b9e-d925e7f051ab\") " Oct 29 22:12:15.763837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.763744 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume" pod "0ed3194e-7db5-40f4-8b9e-d925e7f051ab" (UID: "0ed3194e-7db5-40f4-8b9e-d925e7f051ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:15.763837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.763751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\" (UniqueName: \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\") pod \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\" (UID: \"32046f70-e24b-4ff2-96d8-b5e03b08d5ac\") " Oct 29 22:12:15.764472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.764439 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume" pod "32046f70-e24b-4ff2-96d8-b5e03b08d5ac" (UID: "32046f70-e24b-4ff2-96d8-b5e03b08d5ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:15.764622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.764503 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.768271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.768243 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7" pod "0ed3194e-7db5-40f4-8b9e-d925e7f051ab" (UID: "0ed3194e-7db5-40f4-8b9e-d925e7f051ab"). InnerVolumeSpecName "kube-api-access-bk9h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:15.768383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.768342 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd" pod "32046f70-e24b-4ff2-96d8-b5e03b08d5ac" (UID: "32046f70-e24b-4ff2-96d8-b5e03b08d5ac"). InnerVolumeSpecName "kube-api-access-p8hwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:15.867742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.867649 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hwd\" (UniqueName: \"kubernetes.io/projected/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-kube-api-access-p8hwd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.867742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.867690 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32046f70-e24b-4ff2-96d8-b5e03b08d5ac-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.867742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.867705 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk9h7\" (UniqueName: \"kubernetes.io/projected/0ed3194e-7db5-40f4-8b9e-d925e7f051ab-kube-api-access-bk9h7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:15.993809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:15.993766 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52"] Oct 29 22:12:16.000744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.000714 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a20be1-zsm52"] Oct 29 22:12:16.367637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.367604 4543 generic.go:356] "Generic (PLEG): container finished" podID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" containerID="dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6" exitCode=0 Oct 29 22:12:16.367972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.367649 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" Oct 29 22:12:16.367972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.367684 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" event={"ID":"0ed3194e-7db5-40f4-8b9e-d925e7f051ab","Type":"ContainerDied","Data":"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6"} Oct 29 22:12:16.367972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.367736 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx" event={"ID":"0ed3194e-7db5-40f4-8b9e-d925e7f051ab","Type":"ContainerDied","Data":"4331f2b02e218eee6f5eaf007ef57181aa48d2178658a8a96320797e4b0275d6"} Oct 29 22:12:16.367972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.367765 4543 scope.go:122] "RemoveContainer" containerID="dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6" Oct 29 22:12:16.369604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.369576 4543 generic.go:356] "Generic (PLEG): container finished" podID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" containerID="8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f" exitCode=0 Oct 29 22:12:16.369735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.369620 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" Oct 29 22:12:16.369735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.369652 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" event={"ID":"32046f70-e24b-4ff2-96d8-b5e03b08d5ac","Type":"ContainerDied","Data":"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f"} Oct 29 22:12:16.369735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.369693 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t" event={"ID":"32046f70-e24b-4ff2-96d8-b5e03b08d5ac","Type":"ContainerDied","Data":"ae35a1688a00a33344a254f0c644fdd81f1bf4babd951892bf0e2ed0bba1ae2d"} Oct 29 22:12:16.380598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.380580 4543 scope.go:122] "RemoveContainer" containerID="dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6" Oct 29 22:12:16.380882 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:16.380856 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6\": container with ID starting with dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6 not found: ID does not exist" containerID="dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6" Oct 29 22:12:16.380986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.380891 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6"} err="failed to get container status \"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6\": rpc error: code = NotFound desc = could not find container \"dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6\": container with ID starting with dc8aafb78b0b1fca9499be64bfdd14470fc5e79052dcdbe409cbe6135d8c65b6 not found: ID does not exist" Oct 29 22:12:16.380986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.380911 4543 scope.go:122] "RemoveContainer" containerID="8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f" Oct 29 22:12:16.386977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.386952 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9897a756-8532-4d01-98cf-91df5fafb4b2" path="/var/lib/kubelet/pods/9897a756-8532-4d01-98cf-91df5fafb4b2/volumes" Oct 29 22:12:16.391911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.391892 4543 scope.go:122] "RemoveContainer" containerID="8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f" Oct 29 22:12:16.392187 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:16.392170 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f\": container with ID starting with 8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f not found: ID does not exist" containerID="8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f" Oct 29 22:12:16.392255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.392193 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f"} err="failed to get container status \"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f\": rpc error: code = NotFound desc = could not find container \"8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f\": container with ID starting with 8a0aae4fdaf23637a73d073708065c6264919c399098a1eee9a7df71817a240f not found: ID does not exist" Oct 29 22:12:16.430965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.430934 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:12:16.443861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.443841 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a20be1-85c494496b-khbbx"] Oct 29 22:12:16.473145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.473121 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:12:16.480996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:16.480974 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a20be1-5d897ddcdc-kpr7t"] Oct 29 22:12:17.863366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:17.863330 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:12:17.863920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:17.863888 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" containerID="cri-o://fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507" gracePeriod=30 Oct 29 22:12:18.388093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.388044 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" path="/var/lib/kubelet/pods/0ed3194e-7db5-40f4-8b9e-d925e7f051ab/volumes" Oct 29 22:12:18.388619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.388595 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" path="/var/lib/kubelet/pods/32046f70-e24b-4ff2-96d8-b5e03b08d5ac/volumes" Oct 29 22:12:18.455858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.455826 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww"] Oct 29 22:12:18.456526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456506 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" containerName="payload-processing" Oct 29 22:12:18.456526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456524 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" containerName="payload-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456533 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9897a756-8532-4d01-98cf-91df5fafb4b2" containerName="revoke-keys" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456537 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9897a756-8532-4d01-98cf-91df5fafb4b2" containerName="revoke-keys" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456552 4543 cpu_manager.go:414] "containerMap: removing container" podUID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" containerName="payload-pre-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456557 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" containerName="payload-pre-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456569 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b98860da-1de2-489d-aeaf-e1968915a001" containerName="payload-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456573 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98860da-1de2-489d-aeaf-e1968915a001" containerName="payload-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456579 4543 cpu_manager.go:414] "containerMap: removing container" podUID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" containerName="payload-pre-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456583 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" containerName="payload-pre-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456713 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed3194e-7db5-40f4-8b9e-d925e7f051ab" containerName="payload-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456723 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9897a756-8532-4d01-98cf-91df5fafb4b2" containerName="revoke-keys" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456729 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="32046f70-e24b-4ff2-96d8-b5e03b08d5ac" containerName="payload-pre-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456735 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98860da-1de2-489d-aeaf-e1968915a001" containerName="payload-processing" Oct 29 22:12:18.456765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.456741 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be8328-cdbb-4f7f-8d6e-4ec08bce7af7" containerName="payload-pre-processing" Oct 29 22:12:18.460272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.460253 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.477111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.477057 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:12:18.477577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.477556 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww"] Oct 29 22:12:18.477816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.477794 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:12:18.477906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.477868 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:12:18.484850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.484826 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:12:18.552227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.552192 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.552458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.552424 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.552602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.552583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwd2\" (UniqueName: \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.653988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.653919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.653988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.653985 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.654194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.654055 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwd2\" (UniqueName: \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.654677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.654654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.656867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.656845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.680173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.680152 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwd2\" (UniqueName: \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\") pod \"maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:18.775527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:18.775502 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:19.087818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:19.087790 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww"] Oct 29 22:12:19.088572 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:19.088544 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0024e94_7114_4bd1_8a77_3423aad3a43c.slice/crio-0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355 WatchSource:0}: Error finding container 0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355: Status 404 returned error can't find the container with id 0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355 Oct 29 22:12:19.398720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:19.398690 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" event={"ID":"e0024e94-7114-4bd1-8a77-3423aad3a43c","Type":"ContainerStarted","Data":"0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355"} Oct 29 22:12:20.408120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:20.408020 4543 generic.go:356] "Generic (PLEG): container finished" podID="e0024e94-7114-4bd1-8a77-3423aad3a43c" containerID="000d2b4ecb79961d75d783b9ce99332b49689baf21556592b1f48e6a8e5365ce" exitCode=0 Oct 29 22:12:20.408120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:20.408096 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" event={"ID":"e0024e94-7114-4bd1-8a77-3423aad3a43c","Type":"ContainerDied","Data":"000d2b4ecb79961d75d783b9ce99332b49689baf21556592b1f48e6a8e5365ce"} Oct 29 22:12:21.644576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.644553 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:21.677445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.677392 4543 scope.go:122] "RemoveContainer" containerID="304c2f99c4df189046721ff1f6a0a36283750037909d596a2804538bfaefd422" Oct 29 22:12:21.794374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.794350 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\" (UniqueName: \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\") pod \"e0024e94-7114-4bd1-8a77-3423aad3a43c\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " Oct 29 22:12:21.794521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.794506 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\") pod \"e0024e94-7114-4bd1-8a77-3423aad3a43c\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " Oct 29 22:12:21.794619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.794601 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\") pod \"e0024e94-7114-4bd1-8a77-3423aad3a43c\" (UID: \"e0024e94-7114-4bd1-8a77-3423aad3a43c\") " Oct 29 22:12:21.795026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.794997 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca" pod "e0024e94-7114-4bd1-8a77-3423aad3a43c" (UID: "e0024e94-7114-4bd1-8a77-3423aad3a43c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:21.795502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.795478 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:21.796920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.796893 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2" pod "e0024e94-7114-4bd1-8a77-3423aad3a43c" (UID: "e0024e94-7114-4bd1-8a77-3423aad3a43c"). InnerVolumeSpecName "kube-api-access-6zwd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:21.797066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.797042 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca" pod "e0024e94-7114-4bd1-8a77-3423aad3a43c" (UID: "e0024e94-7114-4bd1-8a77-3423aad3a43c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:21.917099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.909324 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0024e94-7114-4bd1-8a77-3423aad3a43c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:21.917099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:21.909360 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwd2\" (UniqueName: \"kubernetes.io/projected/e0024e94-7114-4bd1-8a77-3423aad3a43c-kube-api-access-6zwd2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:22.432413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:22.432384 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" Oct 29 22:12:22.432413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:22.432397 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww" event={"ID":"e0024e94-7114-4bd1-8a77-3423aad3a43c","Type":"ContainerDied","Data":"0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355"} Oct 29 22:12:22.432593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:22.432439 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af83fd2ef404a6d63129c089cc391b51bf23789a9eeae62c574a82990bee355" Oct 29 22:12:22.744784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:22.744745 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:12:22.745233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:22.744808 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:12:23.137792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.137767 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:12:23.225513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225485 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.225694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.225694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225640 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.225829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225741 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.225829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225770 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:23.225829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225779 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.225829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225815 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.226048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225911 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.226048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.225952 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.226048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.226025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\") pod \"1e455388-f6b8-4b18-b526-cc3807ad7c79\" (UID: \"1e455388-f6b8-4b18-b526-cc3807ad7c79\") " Oct 29 22:12:23.226346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.226318 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:23.227015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.226619 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:23.227015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.226998 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e455388-f6b8-4b18-b526-cc3807ad7c79-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.227407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.227023 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.227407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.227037 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.227407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.227164 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:23.228626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.227627 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:23.230534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.230498 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:23.230625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.230553 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:23.231630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.231597 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:23.232143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.232117 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6" pod "1e455388-f6b8-4b18-b526-cc3807ad7c79" (UID: "1e455388-f6b8-4b18-b526-cc3807ad7c79"). InnerVolumeSpecName "kube-api-access-lvvp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.338955 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvp6\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-kube-api-access-lvvp6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.338993 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.339007 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.339019 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e455388-f6b8-4b18-b526-cc3807ad7c79-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.339029 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.339168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.339041 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e455388-f6b8-4b18-b526-cc3807ad7c79-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:23.445591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.445554 4543 generic.go:356] "Generic (PLEG): container finished" podID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerID="fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507" exitCode=0 Oct 29 22:12:23.445761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.445613 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" Oct 29 22:12:23.445761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.445638 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerDied","Data":"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507"} Oct 29 22:12:23.445761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.445699 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp" event={"ID":"1e455388-f6b8-4b18-b526-cc3807ad7c79","Type":"ContainerDied","Data":"bf02eb19aa7d4736d29fa27dbd6e8f01faaae5b18076b2abfa308d66d01aecd0"} Oct 29 22:12:23.445761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.445726 4543 scope.go:122] "RemoveContainer" containerID="fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507" Oct 29 22:12:23.458449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.458432 4543 scope.go:122] "RemoveContainer" containerID="6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83" Oct 29 22:12:23.468855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.468835 4543 scope.go:122] "RemoveContainer" containerID="fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507" Oct 29 22:12:23.469194 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:23.469172 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507\": container with ID starting with fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507 not found: ID does not exist" containerID="fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507" Oct 29 22:12:23.469258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.469201 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507"} err="failed to get container status \"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507\": rpc error: code = NotFound desc = could not find container \"fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507\": container with ID starting with fb3c4260ba28ffba9f824c3cbfda709e2d2d2861d983c2cf536bf546713fd507 not found: ID does not exist" Oct 29 22:12:23.469258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.469220 4543 scope.go:122] "RemoveContainer" containerID="6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83" Oct 29 22:12:23.469529 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:23.469501 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83\": container with ID starting with 6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83 not found: ID does not exist" containerID="6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83" Oct 29 22:12:23.469578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.469538 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83"} err="failed to get container status \"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83\": rpc error: code = NotFound desc = could not find container \"6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83\": container with ID starting with 6fdb65b005691a44d2b7d5fff482cb9e6e77b80d50d91d44ad4e02b2fc26cf83 not found: ID does not exist" Oct 29 22:12:23.506475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.506453 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:12:23.519649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:23.519627 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-80bd36-openshift-default-9d545f4df-vtqmp"] Oct 29 22:12:24.386903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:24.386865 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" path="/var/lib/kubelet/pods/1e455388-f6b8-4b18-b526-cc3807ad7c79/volumes" Oct 29 22:12:27.038302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.038268 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w"] Oct 29 22:12:27.051228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.051205 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb66b6c-8887w"] Oct 29 22:12:27.138839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.138806 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:12:27.139850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139825 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:27.139984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139852 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:27.139984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139932 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:27.139984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139941 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:27.139984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139974 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e0024e94-7114-4bd1-8a77-3423aad3a43c" containerName="revoke-keys" Oct 29 22:12:27.139984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.139983 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0024e94-7114-4bd1-8a77-3423aad3a43c" containerName="revoke-keys" Oct 29 22:12:27.140288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.140173 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0024e94-7114-4bd1-8a77-3423aad3a43c" containerName="revoke-keys" Oct 29 22:12:27.140288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.140191 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:27.146639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.146616 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.172624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.172595 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:12:27.175679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.173928 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-76c1b7-openshift-default-dockercfg-rvvsj\"" Oct 29 22:12:27.233659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.233628 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.233831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.233766 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.233831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.233797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.233948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.233878 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.234026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.234007 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.234138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.234048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.234245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.234227 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.234304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.234276 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.234372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.234358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgps\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346395 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346469 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346549 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgps\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346713 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346794 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346891 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346949 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.346982 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.373788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.347855 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.380664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.380632 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.381527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.381501 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.381985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.381962 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.391192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.382817 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.391347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.383109 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.391453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.383598 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.423320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.423279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.449099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.433981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgps\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.466483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.466443 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:27.856503 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:27.856471 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb05227c_5018_4052_a277_4864230a5a65.slice/crio-15e1be0cd158d8836e6850d52e73f0a1e9aed03cfb0329412efdc60f2b2ebd86 WatchSource:0}: Error finding container 15e1be0cd158d8836e6850d52e73f0a1e9aed03cfb0329412efdc60f2b2ebd86: Status 404 returned error can't find the container with id 15e1be0cd158d8836e6850d52e73f0a1e9aed03cfb0329412efdc60f2b2ebd86 Oct 29 22:12:27.857008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.856986 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:12:27.859197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.859164 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:27.859277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.859230 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:27.859277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:27.859258 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:27.961219 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:27.961185 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:12:28.387960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.387922 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be832ca-8136-4efe-8cac-167b3298fbed" path="/var/lib/kubelet/pods/9be832ca-8136-4efe-8cac-167b3298fbed/volumes" Oct 29 22:12:28.391783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.391759 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n"] Oct 29 22:12:28.392537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.392522 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e455388-f6b8-4b18-b526-cc3807ad7c79" containerName="istio-proxy" Oct 29 22:12:28.396462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.396438 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.414844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.414822 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n"] Oct 29 22:12:28.422951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.422926 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:12:28.423055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.422989 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:12:28.426365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.426345 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:12:28.426512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.426350 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:12:28.464172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.464149 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.464290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.464215 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw577\" (UniqueName: \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.464428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.464406 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.530197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.530167 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerStarted","Data":"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8"} Oct 29 22:12:28.530318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.530227 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerStarted","Data":"15e1be0cd158d8836e6850d52e73f0a1e9aed03cfb0329412efdc60f2b2ebd86"} Oct 29 22:12:28.565968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.565942 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.566115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.566013 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw577\" (UniqueName: \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.566192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.566137 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.566691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.566671 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.568497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.568478 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.584279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.584234 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podStartSLOduration=1.584221758 podStartE2EDuration="1.584221758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:28.576811678 +0000 UTC m=+1810.318742352" watchObservedRunningTime="2026-10-29 22:12:28.584221758 +0000 UTC m=+1810.326152500" Oct 29 22:12:28.595267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.595246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw577\" (UniqueName: \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\") pod \"maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.717941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.717682 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:28.830977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.828810 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:12:28.860283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:28.860252 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:12:29.063913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.063887 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n"] Oct 29 22:12:29.065968 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:29.065940 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0b2ac8_057d_47f4_823a_8575f7ce4cbe.slice/crio-824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac WatchSource:0}: Error finding container 824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac: Status 404 returned error can't find the container with id 824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac Oct 29 22:12:29.467984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.467959 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:29.473435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.473413 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:29.553519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.553327 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" event={"ID":"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe","Type":"ContainerStarted","Data":"824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac"} Oct 29 22:12:29.553737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.553710 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:29.561097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:29.557238 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:12:30.448374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.448346 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:30.452916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.452899 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.473168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.472974 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-796d71-openshift-default-dockercfg-rrwfs\"" Oct 29 22:12:30.478407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.478384 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:30.511331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511309 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.511472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.511614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511585 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.511716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.511970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511751 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.511970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.511902 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.512120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.512023 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjpg\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.512197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.512175 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.512265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.512240 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.554494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.554466 4543 generic.go:356] "Generic (PLEG): container finished" podID="ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" containerID="c4ceeea9af6bf608424137c7d3e24d5f74b2887b471490f8084a84c80e9260e9" exitCode=0 Oct 29 22:12:30.554644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.554550 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" event={"ID":"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe","Type":"ContainerDied","Data":"c4ceeea9af6bf608424137c7d3e24d5f74b2887b471490f8084a84c80e9260e9"} Oct 29 22:12:30.613271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613216 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613288 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613330 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613356 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjpg\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613428 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613483 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613581 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613629 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.613691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613672 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.614144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.613955 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.614144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.614065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.614285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.614193 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.614579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.614560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.615668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.615635 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.616194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.616165 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.641018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.640993 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.641137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.641061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjpg\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.709261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.709224 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2"] Oct 29 22:12:30.714857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.714840 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.739349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.739328 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2"] Oct 29 22:12:30.766659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.766531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:30.817093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.816277 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.817093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.816505 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdbs\" (UniqueName: \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.817093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.816684 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.918531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.918462 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.918698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.918635 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdbs\" (UniqueName: \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.918795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.918776 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.919641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.919608 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.921158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.921130 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:30.944350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:30.944327 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdbs\" (UniqueName: \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\") pod \"maas-api-revoke-sub-ad3ed25d-4w7m2\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:31.029001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.028971 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:31.108407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.108220 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:31.115527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.115462 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:31.115659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.115558 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:31.115659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.115605 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:31.341987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.341958 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2"] Oct 29 22:12:31.342470 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:31.342448 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec28cfc5_145f_4945_8c66_08b96d4acb3e.slice/crio-2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a WatchSource:0}: Error finding container 2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a: Status 404 returned error can't find the container with id 2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a Oct 29 22:12:31.570163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.570120 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" event={"ID":"ec28cfc5-145f-4945-8c66-08b96d4acb3e","Type":"ContainerStarted","Data":"2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a"} Oct 29 22:12:31.572243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.572211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" event={"ID":"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c","Type":"ContainerStarted","Data":"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394"} Oct 29 22:12:31.572385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.572262 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" event={"ID":"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c","Type":"ContainerStarted","Data":"8d9b0279c97ff86ad68b90f01f60537015b9ee7b7eecb819969cfee7f15d6723"} Oct 29 22:12:31.621337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.620954 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podStartSLOduration=1.620914609 podStartE2EDuration="1.620914609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:31.61999406 +0000 UTC m=+1813.361924738" watchObservedRunningTime="2026-10-29 22:12:31.620914609 +0000 UTC m=+1813.362845283" Oct 29 22:12:31.761448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.761423 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:31.767173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.767154 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:31.844251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.844191 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\" (UniqueName: \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\") pod \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " Oct 29 22:12:31.844352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.844319 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\") pod \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " Oct 29 22:12:31.844533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.844510 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\") pod \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\" (UID: \"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe\") " Oct 29 22:12:31.844669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.844636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca" pod "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" (UID: "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:31.845154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.845133 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.846432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.846412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577" pod "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" (UID: "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe"). InnerVolumeSpecName "kube-api-access-pw577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:31.846908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.846888 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca" pod "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" (UID: "ee0b2ac8-057d-47f4-823a-8575f7ce4cbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:31.946423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.946397 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:31.946423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:31.946421 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw577\" (UniqueName: \"kubernetes.io/projected/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe-kube-api-access-pw577\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:32.322009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.321975 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc"] Oct 29 22:12:32.322732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.322715 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" containerName="revoke-keys" Oct 29 22:12:32.322780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.322735 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" containerName="revoke-keys" Oct 29 22:12:32.322880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.322870 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" containerName="revoke-keys" Oct 29 22:12:32.326635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.326612 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.376975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.376340 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc"] Oct 29 22:12:32.462157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.462115 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.462341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.462315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.462578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.462560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpwm\" (UniqueName: \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.564328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.564246 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpwm\" (UniqueName: \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.564328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.564305 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.564529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.564379 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.565037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.565003 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.566864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.566838 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.582430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.582409 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" Oct 29 22:12:32.582827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.582437 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n" event={"ID":"ee0b2ac8-057d-47f4-823a-8575f7ce4cbe","Type":"ContainerDied","Data":"824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac"} Oct 29 22:12:32.582827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.582485 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="824dadbbe19aeea1ee5d18ba5e76b87c26037fb70c3e404930cdf9179054a0ac" Oct 29 22:12:32.584585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.584401 4543 generic.go:356] "Generic (PLEG): container finished" podID="ec28cfc5-145f-4945-8c66-08b96d4acb3e" containerID="3bac6dd66600b941d29720fd3b418a3e3d02fba9b8f58a5601f23539a153abd0" exitCode=0 Oct 29 22:12:32.584585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.584510 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" event={"ID":"ec28cfc5-145f-4945-8c66-08b96d4acb3e","Type":"ContainerDied","Data":"3bac6dd66600b941d29720fd3b418a3e3d02fba9b8f58a5601f23539a153abd0"} Oct 29 22:12:32.589736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.589715 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpwm\" (UniqueName: \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\") pod \"maas-api-revoke-sub-4b566f05-8k2qc\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.704676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.704648 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:32.773908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.773869 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:32.774101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:32.773938 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:33.176377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.176279 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc"] Oct 29 22:12:33.333110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.332887 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:12:33.364022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.363810 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:12:33.364022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.363932 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.386529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.386498 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-76c1b7-dockercfg-zn8w9\"" Oct 29 22:12:33.387107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.387066 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-76c1b7\"" Oct 29 22:12:33.429917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.428524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.429917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.428581 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4647\" (UniqueName: \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.537113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.533168 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.537113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.533238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4647\" (UniqueName: \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.537113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.535294 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.562983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.562948 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4647\" (UniqueName: \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\") pod \"payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.595421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.595392 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" event={"ID":"a48a0788-2494-4f53-bd4d-fd01e3da7947","Type":"ContainerStarted","Data":"2a1426a23f14f7e133aed2caf83ffe8b95c67f6f46ae930f321fd5dd711b077a"} Oct 29 22:12:33.693823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.693792 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:33.768419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.767976 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:33.768419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.768030 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:33.862102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.860665 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" podUID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" containerName="payload-processing" containerID="cri-o://ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1" gracePeriod=25 Oct 29 22:12:33.862102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.861380 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" podUID="4aa56d71-090c-4ba2-be75-cace1ab71f58" containerName="payload-pre-processing" containerID="cri-o://9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318" gracePeriod=25 Oct 29 22:12:33.875133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.875055 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:33.963232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.963206 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\") pod \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " Oct 29 22:12:33.963484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.963469 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\" (UniqueName: \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\") pod \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " Oct 29 22:12:33.963540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.963494 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\") pod \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\" (UID: \"ec28cfc5-145f-4945-8c66-08b96d4acb3e\") " Oct 29 22:12:33.964739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.964714 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca" pod "ec28cfc5-145f-4945-8c66-08b96d4acb3e" (UID: "ec28cfc5-145f-4945-8c66-08b96d4acb3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:33.967045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.967010 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca" pod "ec28cfc5-145f-4945-8c66-08b96d4acb3e" (UID: "ec28cfc5-145f-4945-8c66-08b96d4acb3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:33.967376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:33.967357 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs" pod "ec28cfc5-145f-4945-8c66-08b96d4acb3e" (UID: "ec28cfc5-145f-4945-8c66-08b96d4acb3e"). InnerVolumeSpecName "kube-api-access-cbdbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:34.065441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.064374 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:12:34.065441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.065349 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdbs\" (UniqueName: \"kubernetes.io/projected/ec28cfc5-145f-4945-8c66-08b96d4acb3e-kube-api-access-cbdbs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.065441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.065379 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.065441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.065393 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec28cfc5-145f-4945-8c66-08b96d4acb3e-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.184934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.184893 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:12:34.267562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.266999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\") pod \"4aa56d71-090c-4ba2-be75-cace1ab71f58\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " Oct 29 22:12:34.267562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.267040 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\" (UniqueName: \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\") pod \"4aa56d71-090c-4ba2-be75-cace1ab71f58\" (UID: \"4aa56d71-090c-4ba2-be75-cace1ab71f58\") " Oct 29 22:12:34.267562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.267452 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume" pod "4aa56d71-090c-4ba2-be75-cace1ab71f58" (UID: "4aa56d71-090c-4ba2-be75-cace1ab71f58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:34.267562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.267538 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa56d71-090c-4ba2-be75-cace1ab71f58-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.269679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.269656 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9" pod "4aa56d71-090c-4ba2-be75-cace1ab71f58" (UID: "4aa56d71-090c-4ba2-be75-cace1ab71f58"). InnerVolumeSpecName "kube-api-access-v8dk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:34.275409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.275388 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:12:34.368374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.368345 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\") pod \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " Oct 29 22:12:34.368588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.368432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\" (UniqueName: \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\") pod \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\" (UID: \"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a\") " Oct 29 22:12:34.368843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.368796 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume" pod "e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" (UID: "e4e8e008-e789-4c40-87b6-2c1ee0c5b91a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:34.369047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.369028 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.369149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.369055 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dk9\" (UniqueName: \"kubernetes.io/projected/4aa56d71-090c-4ba2-be75-cace1ab71f58-kube-api-access-v8dk9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.371439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.371411 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c" pod "e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" (UID: "e4e8e008-e789-4c40-87b6-2c1ee0c5b91a"). InnerVolumeSpecName "kube-api-access-9qj6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:34.470019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.469993 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qj6c\" (UniqueName: \"kubernetes.io/projected/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a-kube-api-access-9qj6c\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:34.605047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.604972 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" event={"ID":"a48a0788-2494-4f53-bd4d-fd01e3da7947","Type":"ContainerStarted","Data":"11241233380ef32ebac3d39a613f8bb7171a0a7e6dced195178e9d6b34e86df8"} Oct 29 22:12:34.606553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.606526 4543 generic.go:356] "Generic (PLEG): container finished" podID="4aa56d71-090c-4ba2-be75-cace1ab71f58" containerID="9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318" exitCode=0 Oct 29 22:12:34.606674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.606565 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" Oct 29 22:12:34.606674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.606615 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" event={"ID":"4aa56d71-090c-4ba2-be75-cace1ab71f58","Type":"ContainerDied","Data":"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318"} Oct 29 22:12:34.606674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.606665 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r" event={"ID":"4aa56d71-090c-4ba2-be75-cace1ab71f58","Type":"ContainerDied","Data":"681816e9f9dcb18a69d4f37fecb31665bd19aa18540cdd5cb73808e89247eb67"} Oct 29 22:12:34.606837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.606692 4543 scope.go:122] "RemoveContainer" containerID="9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318" Oct 29 22:12:34.608679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.608651 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" event={"ID":"94f25693-735f-44db-852b-c78561e828de","Type":"ContainerStarted","Data":"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26"} Oct 29 22:12:34.608792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.608705 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" event={"ID":"94f25693-735f-44db-852b-c78561e828de","Type":"ContainerStarted","Data":"809a09f77ebf8bda369b7dfc07921e60310b735383dee234b9f24f0a8126bb24"} Oct 29 22:12:34.608792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.608738 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:34.610317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.610291 4543 generic.go:356] "Generic (PLEG): container finished" podID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" containerID="ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1" exitCode=0 Oct 29 22:12:34.610479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.610329 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" Oct 29 22:12:34.610479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.610363 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" event={"ID":"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a","Type":"ContainerDied","Data":"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1"} Oct 29 22:12:34.610479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.610393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m" event={"ID":"e4e8e008-e789-4c40-87b6-2c1ee0c5b91a","Type":"ContainerDied","Data":"4b5b962247ede1e03aea34f841f7bf5d2c75ca6686c5ed7ef00027430f92467c"} Oct 29 22:12:34.612389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.612334 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" event={"ID":"ec28cfc5-145f-4945-8c66-08b96d4acb3e","Type":"ContainerDied","Data":"2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a"} Oct 29 22:12:34.612389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.612378 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2141a0f6c3fd05723f1c6fd339f17eb50e39da66d066bb28d0fd83f0461e0c9a" Oct 29 22:12:34.612562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.612444 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2" Oct 29 22:12:34.620723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.620702 4543 scope.go:122] "RemoveContainer" containerID="9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318" Oct 29 22:12:34.621000 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:34.620981 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318\": container with ID starting with 9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318 not found: ID does not exist" containerID="9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318" Oct 29 22:12:34.621095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.621010 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318"} err="failed to get container status \"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318\": rpc error: code = NotFound desc = could not find container \"9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318\": container with ID starting with 9af9307e10b6cbbfe54e3ac31f21655c8472b383d63dacb82e6a836b885f1318 not found: ID does not exist" Oct 29 22:12:34.621095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.621035 4543 scope.go:122] "RemoveContainer" containerID="ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1" Oct 29 22:12:34.714319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.714274 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" podStartSLOduration=2.714261419 podStartE2EDuration="2.714261419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:34.655291876 +0000 UTC m=+1816.397222550" watchObservedRunningTime="2026-10-29 22:12:34.714261419 +0000 UTC m=+1816.456192091" Oct 29 22:12:34.716315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.716291 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:12:34.727725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.722573 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-665b09-64868f6f4-qnj7m"] Oct 29 22:12:34.786554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.782393 4543 scope.go:122] "RemoveContainer" containerID="ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1" Oct 29 22:12:34.786554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.782879 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:34.786554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.782923 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:34.786554 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:34.783273 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1\": container with ID starting with ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1 not found: ID does not exist" containerID="ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1" Oct 29 22:12:34.786554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.783299 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1"} err="failed to get container status \"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1\": rpc error: code = NotFound desc = could not find container \"ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1\": container with ID starting with ce24a839f08f85d2d70e0b401e722c3a91ff1d1e65184194ea413bb420a64ff1 not found: ID does not exist" Oct 29 22:12:34.862170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.848313 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" podStartSLOduration=1.848291839 podStartE2EDuration="1.848291839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:34.78218958 +0000 UTC m=+1816.524120254" watchObservedRunningTime="2026-10-29 22:12:34.848291839 +0000 UTC m=+1816.590222514" Oct 29 22:12:34.868573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.868542 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:12:34.877755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:34.877727 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-665b09-c77f8b67f-gfd5r"] Oct 29 22:12:35.622838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:35.622809 4543 generic.go:356] "Generic (PLEG): container finished" podID="a48a0788-2494-4f53-bd4d-fd01e3da7947" containerID="11241233380ef32ebac3d39a613f8bb7171a0a7e6dced195178e9d6b34e86df8" exitCode=0 Oct 29 22:12:35.623279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:35.622893 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" event={"ID":"a48a0788-2494-4f53-bd4d-fd01e3da7947","Type":"ContainerDied","Data":"11241233380ef32ebac3d39a613f8bb7171a0a7e6dced195178e9d6b34e86df8"} Oct 29 22:12:35.767364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:35.767330 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:35.767535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:35.767380 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:36.386743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:36.386709 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa56d71-090c-4ba2-be75-cace1ab71f58" path="/var/lib/kubelet/pods/4aa56d71-090c-4ba2-be75-cace1ab71f58/volumes" Oct 29 22:12:36.387194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:36.387179 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" path="/var/lib/kubelet/pods/e4e8e008-e789-4c40-87b6-2c1ee0c5b91a/volumes" Oct 29 22:12:36.820833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:36.776740 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:36.878101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:36.776795 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:36.981948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:36.981923 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:37.081964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.081936 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\") pod \"a48a0788-2494-4f53-bd4d-fd01e3da7947\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " Oct 29 22:12:37.082196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.082025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\") pod \"a48a0788-2494-4f53-bd4d-fd01e3da7947\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " Oct 29 22:12:37.082196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.082176 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\" (UniqueName: \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\") pod \"a48a0788-2494-4f53-bd4d-fd01e3da7947\" (UID: \"a48a0788-2494-4f53-bd4d-fd01e3da7947\") " Oct 29 22:12:37.082397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.082370 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca" pod "a48a0788-2494-4f53-bd4d-fd01e3da7947" (UID: "a48a0788-2494-4f53-bd4d-fd01e3da7947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:37.082698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.082677 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:37.084503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.084477 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca" pod "a48a0788-2494-4f53-bd4d-fd01e3da7947" (UID: "a48a0788-2494-4f53-bd4d-fd01e3da7947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:37.085521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.085500 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm" pod "a48a0788-2494-4f53-bd4d-fd01e3da7947" (UID: "a48a0788-2494-4f53-bd4d-fd01e3da7947"). InnerVolumeSpecName "kube-api-access-qnpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:37.183888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.183823 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48a0788-2494-4f53-bd4d-fd01e3da7947-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:37.183888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.183850 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnpwm\" (UniqueName: \"kubernetes.io/projected/a48a0788-2494-4f53-bd4d-fd01e3da7947-kube-api-access-qnpwm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:37.676929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.676904 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" Oct 29 22:12:37.677135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.676967 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc" event={"ID":"a48a0788-2494-4f53-bd4d-fd01e3da7947","Type":"ContainerDied","Data":"2a1426a23f14f7e133aed2caf83ffe8b95c67f6f46ae930f321fd5dd711b077a"} Oct 29 22:12:37.677135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.677016 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a1426a23f14f7e133aed2caf83ffe8b95c67f6f46ae930f321fd5dd711b077a" Oct 29 22:12:37.767722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.767686 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 29 22:12:37.767895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:37.767736 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 29 22:12:38.822734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:38.822701 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:38.829604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:38.829580 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:38.857124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:38.856964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:39.051741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:39.051708 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:12:39.094877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:39.094797 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:12:43.520316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:43.520282 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww"] Oct 29 22:12:43.556974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:43.556946 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-665b09-zbjww"] Oct 29 22:12:44.053205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.053162 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" podUID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" containerName="payload-processing" containerID="cri-o://72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159" gracePeriod=25 Oct 29 22:12:44.095952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.095912 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" podUID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" containerName="payload-pre-processing" containerID="cri-o://ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b" gracePeriod=25 Oct 29 22:12:44.391342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.391307 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0024e94-7114-4bd1-8a77-3423aad3a43c" path="/var/lib/kubelet/pods/e0024e94-7114-4bd1-8a77-3423aad3a43c/volumes" Oct 29 22:12:44.444167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.444141 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:12:44.563918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.563888 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\" (UniqueName: \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\") pod \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " Oct 29 22:12:44.564375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.563936 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\") pod \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\" (UID: \"dafde8e3-a87c-42dc-9189-e9a8a1043a40\") " Oct 29 22:12:44.565505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.565479 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume" pod "dafde8e3-a87c-42dc-9189-e9a8a1043a40" (UID: "dafde8e3-a87c-42dc-9189-e9a8a1043a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:44.567369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.567340 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx" pod "dafde8e3-a87c-42dc-9189-e9a8a1043a40" (UID: "dafde8e3-a87c-42dc-9189-e9a8a1043a40"). InnerVolumeSpecName "kube-api-access-ks4jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:44.569688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.569673 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:12:44.669185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.669106 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4jx\" (UniqueName: \"kubernetes.io/projected/dafde8e3-a87c-42dc-9189-e9a8a1043a40-kube-api-access-ks4jx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:44.669384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.669366 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafde8e3-a87c-42dc-9189-e9a8a1043a40-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:44.768436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.768400 4543 generic.go:356] "Generic (PLEG): container finished" podID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" containerID="ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b" exitCode=0 Oct 29 22:12:44.768647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.768465 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" Oct 29 22:12:44.768647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.768478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" event={"ID":"e9982b2b-afc7-4802-aebb-98b3dba4d46b","Type":"ContainerDied","Data":"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b"} Oct 29 22:12:44.768647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.768567 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8" event={"ID":"e9982b2b-afc7-4802-aebb-98b3dba4d46b","Type":"ContainerDied","Data":"c7d9c291cf2b4734a3b9e98a98e9329489d30d782b9fcb554c42c54aa65867ac"} Oct 29 22:12:44.768647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.768593 4543 scope.go:122] "RemoveContainer" containerID="ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b" Oct 29 22:12:44.770118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.770056 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\") pod \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " Oct 29 22:12:44.770297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.770273 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\" (UniqueName: \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\") pod \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\" (UID: \"e9982b2b-afc7-4802-aebb-98b3dba4d46b\") " Oct 29 22:12:44.770587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.770567 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume" pod "e9982b2b-afc7-4802-aebb-98b3dba4d46b" (UID: "e9982b2b-afc7-4802-aebb-98b3dba4d46b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:44.771129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.771105 4543 generic.go:356] "Generic (PLEG): container finished" podID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" containerID="72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159" exitCode=0 Oct 29 22:12:44.771266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.771149 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9982b2b-afc7-4802-aebb-98b3dba4d46b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:44.771266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.771188 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" event={"ID":"dafde8e3-a87c-42dc-9189-e9a8a1043a40","Type":"ContainerDied","Data":"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159"} Oct 29 22:12:44.771266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.771218 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" Oct 29 22:12:44.771266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.771222 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l" event={"ID":"dafde8e3-a87c-42dc-9189-e9a8a1043a40","Type":"ContainerDied","Data":"8d908901378b0e1b2e52a73a6af27fede1f2857862c1f97ed666bbcabfa8cad7"} Oct 29 22:12:44.772745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.772718 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz" pod "e9982b2b-afc7-4802-aebb-98b3dba4d46b" (UID: "e9982b2b-afc7-4802-aebb-98b3dba4d46b"). InnerVolumeSpecName "kube-api-access-945tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:44.783793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.783775 4543 scope.go:122] "RemoveContainer" containerID="ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b" Oct 29 22:12:44.784053 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:44.784031 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b\": container with ID starting with ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b not found: ID does not exist" containerID="ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b" Oct 29 22:12:44.784135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.784061 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b"} err="failed to get container status \"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b\": rpc error: code = NotFound desc = could not find container \"ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b\": container with ID starting with ca1173204ba4861fe67a2bda19cb2adf3aba44e27dcfa3dc1c7dfd400288c05b not found: ID does not exist" Oct 29 22:12:44.784135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.784096 4543 scope.go:122] "RemoveContainer" containerID="72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159" Oct 29 22:12:44.796778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.796757 4543 scope.go:122] "RemoveContainer" containerID="72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159" Oct 29 22:12:44.797128 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:44.797061 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159\": container with ID starting with 72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159 not found: ID does not exist" containerID="72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159" Oct 29 22:12:44.797248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.797138 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159"} err="failed to get container status \"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159\": rpc error: code = NotFound desc = could not find container \"72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159\": container with ID starting with 72f76897a07bacf0c8aa0ad6616b0ba759061546e55190e254e69af2e35e4159 not found: ID does not exist" Oct 29 22:12:44.858163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.858129 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:12:44.862637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.862610 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7721b3-7d9b686f5b-q2n2l"] Oct 29 22:12:44.871961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.871938 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945tz\" (UniqueName: \"kubernetes.io/projected/e9982b2b-afc7-4802-aebb-98b3dba4d46b-kube-api-access-945tz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:44.911681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.911655 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:12:44.912748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912727 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ec28cfc5-145f-4945-8c66-08b96d4acb3e" containerName="revoke-keys" Oct 29 22:12:44.912794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912754 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec28cfc5-145f-4945-8c66-08b96d4acb3e" containerName="revoke-keys" Oct 29 22:12:44.912837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912798 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" containerName="payload-processing" Oct 29 22:12:44.912837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912807 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" containerName="payload-processing" Oct 29 22:12:44.912905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912836 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4aa56d71-090c-4ba2-be75-cace1ab71f58" containerName="payload-pre-processing" Oct 29 22:12:44.912905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912845 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa56d71-090c-4ba2-be75-cace1ab71f58" containerName="payload-pre-processing" Oct 29 22:12:44.912905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912890 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" containerName="payload-pre-processing" Oct 29 22:12:44.912905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912898 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" containerName="payload-pre-processing" Oct 29 22:12:44.913023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912915 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a48a0788-2494-4f53-bd4d-fd01e3da7947" containerName="revoke-keys" Oct 29 22:12:44.913023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912922 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48a0788-2494-4f53-bd4d-fd01e3da7947" containerName="revoke-keys" Oct 29 22:12:44.913023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912934 4543 cpu_manager.go:414] "containerMap: removing container" podUID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" containerName="payload-processing" Oct 29 22:12:44.913023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.912941 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" containerName="payload-processing" Oct 29 22:12:44.913175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913126 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48a0788-2494-4f53-bd4d-fd01e3da7947" containerName="revoke-keys" Oct 29 22:12:44.913175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913140 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" containerName="payload-processing" Oct 29 22:12:44.913270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913197 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec28cfc5-145f-4945-8c66-08b96d4acb3e" containerName="revoke-keys" Oct 29 22:12:44.913270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913226 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e8e008-e789-4c40-87b6-2c1ee0c5b91a" containerName="payload-processing" Oct 29 22:12:44.913270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913237 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa56d71-090c-4ba2-be75-cace1ab71f58" containerName="payload-pre-processing" Oct 29 22:12:44.913270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.913256 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" containerName="payload-pre-processing" Oct 29 22:12:44.918614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.918537 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.956640 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bef83b-openshift-default-dockercfg-hrrzn\"" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973456 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4nn\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973549 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973631 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973684 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973710 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973746 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973850 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973896 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.975097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.973929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:44.979127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:44.978280 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075059 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075171 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075240 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075358 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075408 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075440 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4nn\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.075611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.078102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.076128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.087153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.086442 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.108123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.087770 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.108123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.088036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.108123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.088421 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.119175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.118201 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.119175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.118531 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.167144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.163886 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4nn\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.178201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.174133 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.296157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.291417 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:45.301066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.300221 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:12:45.302919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.302886 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7721b3-6b766f7579-zljh8"] Oct 29 22:12:45.663147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.663116 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:12:45.667897 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:45.667873 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6b5abd_9192_4a00_9ab5_566bcd01dde7.slice/crio-c365366b573257b0400bec8e7320bf8469cddd703724d41963be30787464b3b5 WatchSource:0}: Error finding container c365366b573257b0400bec8e7320bf8469cddd703724d41963be30787464b3b5: Status 404 returned error can't find the container with id c365366b573257b0400bec8e7320bf8469cddd703724d41963be30787464b3b5 Oct 29 22:12:45.669695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.669674 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:12:45.672310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.672280 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:45.672416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.672379 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:45.672490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.672449 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:45.832546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.825246 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerStarted","Data":"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59"} Oct 29 22:12:45.832546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.825384 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerStarted","Data":"c365366b573257b0400bec8e7320bf8469cddd703724d41963be30787464b3b5"} Oct 29 22:12:45.899956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:45.899901 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podStartSLOduration=1.899887527 podStartE2EDuration="1.899887527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:45.896245292 +0000 UTC m=+1827.638175995" watchObservedRunningTime="2026-10-29 22:12:45.899887527 +0000 UTC m=+1827.641818200" Oct 29 22:12:46.204749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.204711 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7"] Oct 29 22:12:46.211036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.210313 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.226271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.226244 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdz8\" (UniqueName: \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.226453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.226416 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.226578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.226561 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.226887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.226869 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7"] Oct 29 22:12:46.228184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.228161 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:12:46.228575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.228540 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:12:46.279469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.278175 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:12:46.279469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.278175 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:12:46.293099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.293050 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:46.299457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.299431 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:46.339761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.328152 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdz8\" (UniqueName: \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.340129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.340055 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.340277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.340199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.341332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.341305 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.344124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.344100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.364927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.364903 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdz8\" (UniqueName: \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\") pod \"maas-api-revoke-sub-dbd0e78e-fgft7\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.387428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.387399 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafde8e3-a87c-42dc-9189-e9a8a1043a40" path="/var/lib/kubelet/pods/dafde8e3-a87c-42dc-9189-e9a8a1043a40/volumes" Oct 29 22:12:46.387913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.387897 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9982b2b-afc7-4802-aebb-98b3dba4d46b" path="/var/lib/kubelet/pods/e9982b2b-afc7-4802-aebb-98b3dba4d46b/volumes" Oct 29 22:12:46.532982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.532953 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:46.854040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.854015 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7"] Oct 29 22:12:46.854683 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:46.854655 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb48345_5272_4cde_8476_7d41edc3e788.slice/crio-2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5 WatchSource:0}: Error finding container 2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5: Status 404 returned error can't find the container with id 2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5 Oct 29 22:12:46.887656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.887624 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" event={"ID":"fcb48345-5272-4cde-8476-7d41edc3e788","Type":"ContainerStarted","Data":"2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5"} Oct 29 22:12:46.887948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.887926 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:46.888949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:46.888928 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:12:47.040239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:47.040210 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t"] Oct 29 22:12:47.047531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:47.047460 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b475f6-qbt2t"] Oct 29 22:12:47.900248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:47.900217 4543 generic.go:356] "Generic (PLEG): container finished" podID="fcb48345-5272-4cde-8476-7d41edc3e788" containerID="eeb1ffeefdeb4d53e83092580310165d3e73fe416bfa13dd8a64b25c82d16e4b" exitCode=0 Oct 29 22:12:47.900680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:47.900301 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" event={"ID":"fcb48345-5272-4cde-8476-7d41edc3e788","Type":"ContainerDied","Data":"eeb1ffeefdeb4d53e83092580310165d3e73fe416bfa13dd8a64b25c82d16e4b"} Oct 29 22:12:48.388852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:48.388803 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e155b22-7abe-42d4-9100-ee19e3083836" path="/var/lib/kubelet/pods/3e155b22-7abe-42d4-9100-ee19e3083836/volumes" Oct 29 22:12:49.213606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.213578 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:49.315821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.315777 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\") pod \"fcb48345-5272-4cde-8476-7d41edc3e788\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " Oct 29 22:12:49.316014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.315854 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\" (UniqueName: \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\") pod \"fcb48345-5272-4cde-8476-7d41edc3e788\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " Oct 29 22:12:49.316014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.315944 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\") pod \"fcb48345-5272-4cde-8476-7d41edc3e788\" (UID: \"fcb48345-5272-4cde-8476-7d41edc3e788\") " Oct 29 22:12:49.317044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.316359 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca" pod "fcb48345-5272-4cde-8476-7d41edc3e788" (UID: "fcb48345-5272-4cde-8476-7d41edc3e788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:49.317044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.317011 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:49.318555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.318528 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8" pod "fcb48345-5272-4cde-8476-7d41edc3e788" (UID: "fcb48345-5272-4cde-8476-7d41edc3e788"). InnerVolumeSpecName "kube-api-access-qwdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:49.319064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.319030 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca" pod "fcb48345-5272-4cde-8476-7d41edc3e788" (UID: "fcb48345-5272-4cde-8476-7d41edc3e788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:49.418543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.418476 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb48345-5272-4cde-8476-7d41edc3e788-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:49.418543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.418511 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdz8\" (UniqueName: \"kubernetes.io/projected/fcb48345-5272-4cde-8476-7d41edc3e788-kube-api-access-qwdz8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:49.717610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.717578 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:49.717847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.717821 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" containerID="cri-o://143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394" gracePeriod=30 Oct 29 22:12:49.993291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.993206 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" event={"ID":"fcb48345-5272-4cde-8476-7d41edc3e788","Type":"ContainerDied","Data":"2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5"} Oct 29 22:12:49.993508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.993485 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7" Oct 29 22:12:49.993644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:49.993629 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b32190c20a5dca3441ca6732506c29bc270672524c86812237a2d4b533098d5" Oct 29 22:12:52.015012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.014977 4543 generic.go:356] "Generic (PLEG): container finished" podID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerID="6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932" exitCode=0 Oct 29 22:12:52.015573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.015053 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerDied","Data":"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932"} Oct 29 22:12:52.015573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.015529 4543 scope.go:122] "RemoveContainer" containerID="6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932" Oct 29 22:12:52.017107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.017054 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:52.017213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.017142 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:52.017213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:52.017181 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:12:53.108097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.107964 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerStarted","Data":"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4"} Oct 29 22:12:53.616447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.616411 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:12:53.617287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.617256 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 29 22:12:53.617440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.617319 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 29 22:12:53.853886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.853845 4543 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:12:53.854132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:53.853925 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:12:54.306305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.306274 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:12:54.417187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.417157 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:12:54.417906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.417884 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fcb48345-5272-4cde-8476-7d41edc3e788" containerName="revoke-keys" Oct 29 22:12:54.417906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.417907 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb48345-5272-4cde-8476-7d41edc3e788" containerName="revoke-keys" Oct 29 22:12:54.418124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.418098 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb48345-5272-4cde-8476-7d41edc3e788" containerName="revoke-keys" Oct 29 22:12:54.420923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.420903 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.460610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.460582 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:12:54.467126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.467100 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-15154b-dockercfg-5p6t5\"" Oct 29 22:12:54.467322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.467113 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-15154b\"" Oct 29 22:12:54.478771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.478723 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:12:54.516008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.515885 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.592303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.585102 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4cp\" (UniqueName: \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.592303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.585364 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.677097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.664776 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 29 22:12:54.677097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.664849 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 29 22:12:54.752215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.752182 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4cp\" (UniqueName: \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.780094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.777688 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.780094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.777753 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hq8\" (UniqueName: \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.780094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.777862 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.780094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.770896 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:12:54.780094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.778636 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.809212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.808692 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4cp\" (UniqueName: \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\") pod \"payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:54.879396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.879317 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.879545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.879398 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hq8\" (UniqueName: \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.880592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.880569 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:54.908793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:54.908737 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hq8\" (UniqueName: \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\") pod \"payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:55.006647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.006622 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:55.010036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.010021 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:55.061629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.061598 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.081762 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.081862 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.081905 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.081958 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.082005 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.082138 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.082269 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.082287 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.082325 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\") pod \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\" (UID: \"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c\") " Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.083537 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.083666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.083756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.083906 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.084110 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.086291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.084512 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.088489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.087624 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.088489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.087786 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "kube-api-access-mvjpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.088489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.088210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo" pod "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" (UID: "f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:55.170089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.165138 4543 generic.go:356] "Generic (PLEG): container finished" podID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerID="143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394" exitCode=0 Oct 29 22:12:55.170089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.166313 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" Oct 29 22:12:55.170089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.168393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" event={"ID":"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c","Type":"ContainerDied","Data":"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394"} Oct 29 22:12:55.170089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.168447 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp" event={"ID":"f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c","Type":"ContainerDied","Data":"8d9b0279c97ff86ad68b90f01f60537015b9ee7b7eecb819969cfee7f15d6723"} Oct 29 22:12:55.170089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.168477 4543 scope.go:122] "RemoveContainer" containerID="143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394" Oct 29 22:12:55.184166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184146 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184298 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184411 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184513 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184608 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184704 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184790 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184876 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjpg\" (UniqueName: \"kubernetes.io/projected/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-kube-api-access-mvjpg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.184944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.184937 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.188322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.188301 4543 scope.go:122] "RemoveContainer" containerID="143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394" Oct 29 22:12:55.188680 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:55.188651 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394\": container with ID starting with 143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394 not found: ID does not exist" containerID="143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394" Oct 29 22:12:55.188750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.188725 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394"} err="failed to get container status \"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394\": rpc error: code = NotFound desc = could not find container \"143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394\": container with ID starting with 143bbfb660499bf8d80c48e0de78e6d7025c563498f315214ee8d8bb91b5f394 not found: ID does not exist" Oct 29 22:12:55.266321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.266291 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:55.271878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.271850 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-q4pmp"] Oct 29 22:12:55.399340 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:55.398571 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c6e527_f6e0_4dae_8b3b_2a30cd9c6fff.slice/crio-54496cb6f9fb2e8527685c86728a1f9f8224d73cf0e1915ccd773de8b7b4f4ea WatchSource:0}: Error finding container 54496cb6f9fb2e8527685c86728a1f9f8224d73cf0e1915ccd773de8b7b4f4ea: Status 404 returned error can't find the container with id 54496cb6f9fb2e8527685c86728a1f9f8224d73cf0e1915ccd773de8b7b4f4ea Oct 29 22:12:55.400388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.400363 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:12:55.436436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.436407 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:12:55.465365 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:55.463211 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7402b9b5_9920_4800_82d1_d35f3f0b0426.slice/crio-e046f3abeacb72cc0799d597d05d48a23ea91bdcc5bce99cb2fde1ae22afedaa WatchSource:0}: Error finding container e046f3abeacb72cc0799d597d05d48a23ea91bdcc5bce99cb2fde1ae22afedaa: Status 404 returned error can't find the container with id e046f3abeacb72cc0799d597d05d48a23ea91bdcc5bce99cb2fde1ae22afedaa Oct 29 22:12:55.616165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.616137 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 29 22:12:55.616322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.616184 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 29 22:12:55.660616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.660582 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 22:12:55.660986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.660812 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" containerID="cri-o://1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b" gracePeriod=30 Oct 29 22:12:55.905904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.905869 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4"] Oct 29 22:12:55.906569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.906555 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" Oct 29 22:12:55.906634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.906571 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" Oct 29 22:12:55.906741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.906730 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" containerName="istio-proxy" Oct 29 22:12:55.909585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.909569 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:55.927144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.927122 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:12:55.927264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.927195 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:12:55.927446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.927433 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:12:55.927603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.927592 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:12:55.931619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:55.931584 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4"] Oct 29 22:12:56.006173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.006143 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7gm\" (UniqueName: \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.006442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.006421 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.006488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.006469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.107473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.107402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7gm\" (UniqueName: \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.107679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.107657 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.107737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.107721 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.108295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.108272 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.109962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.109938 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.154948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.154919 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7gm\" (UniqueName: \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.175673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.175641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" event={"ID":"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff","Type":"ContainerStarted","Data":"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9"} Oct 29 22:12:56.175798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.175705 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" event={"ID":"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff","Type":"ContainerStarted","Data":"54496cb6f9fb2e8527685c86728a1f9f8224d73cf0e1915ccd773de8b7b4f4ea"} Oct 29 22:12:56.175887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.175875 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:12:56.178827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.178806 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" event={"ID":"7402b9b5-9920-4800-82d1-d35f3f0b0426","Type":"ContainerStarted","Data":"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b"} Oct 29 22:12:56.178922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.178843 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" event={"ID":"7402b9b5-9920-4800-82d1-d35f3f0b0426","Type":"ContainerStarted","Data":"e046f3abeacb72cc0799d597d05d48a23ea91bdcc5bce99cb2fde1ae22afedaa"} Oct 29 22:12:56.179002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.178989 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:12:56.222967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.222950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:56.255110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.255043 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" podStartSLOduration=2.255030414 podStartE2EDuration="2.255030414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:56.251306907 +0000 UTC m=+1837.993237581" watchObservedRunningTime="2026-10-29 22:12:56.255030414 +0000 UTC m=+1837.996961087" Oct 29 22:12:56.310255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.310197 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" podStartSLOduration=2.310179325 podStartE2EDuration="2.310179325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:56.302417904 +0000 UTC m=+1838.044348576" watchObservedRunningTime="2026-10-29 22:12:56.310179325 +0000 UTC m=+1838.052109999" Oct 29 22:12:56.386530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.386458 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c" path="/var/lib/kubelet/pods/f41a71a0-9d0b-4bc7-b850-7e21e8a4a28c/volumes" Oct 29 22:12:56.567815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.567770 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4"] Oct 29 22:12:56.568760 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:12:56.568731 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00329f2b_90fc_4c81_8323_9f3a415a7c8e.slice/crio-8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767 WatchSource:0}: Error finding container 8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767: Status 404 returned error can't find the container with id 8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767 Oct 29 22:12:56.615743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.615719 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 29 22:12:56.615855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:56.615765 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 29 22:12:57.189418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:57.189325 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" event={"ID":"00329f2b-90fc-4c81-8323-9f3a415a7c8e","Type":"ContainerStarted","Data":"8208c8813e6c0ff3c10fd60827ff175916ead06b14231555550ce08dd6200b90"} Oct 29 22:12:57.189418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:57.189381 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" event={"ID":"00329f2b-90fc-4c81-8323-9f3a415a7c8e","Type":"ContainerStarted","Data":"8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767"} Oct 29 22:12:57.250102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:57.245439 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" podStartSLOduration=2.245420391 podStartE2EDuration="2.245420391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:57.243414566 +0000 UTC m=+1838.985345239" watchObservedRunningTime="2026-10-29 22:12:57.245420391 +0000 UTC m=+1838.987351066" Oct 29 22:12:57.616532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:57.616502 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 29 22:12:57.616932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:57.616551 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 29 22:12:57.980139 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:12:57.980063 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:12:58.202088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:58.202037 4543 generic.go:356] "Generic (PLEG): container finished" podID="00329f2b-90fc-4c81-8323-9f3a415a7c8e" containerID="8208c8813e6c0ff3c10fd60827ff175916ead06b14231555550ce08dd6200b90" exitCode=0 Oct 29 22:12:58.202352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:58.202110 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" event={"ID":"00329f2b-90fc-4c81-8323-9f3a415a7c8e","Type":"ContainerDied","Data":"8208c8813e6c0ff3c10fd60827ff175916ead06b14231555550ce08dd6200b90"} Oct 29 22:12:58.674373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:58.674334 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:12:58.676234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:58.676190 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:12:59.377169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.377133 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:12:59.483861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.483835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\") pod \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " Oct 29 22:12:59.484050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.484033 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\") pod \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " Oct 29 22:12:59.484173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.484133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\" (UniqueName: \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\") pod \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\" (UID: \"00329f2b-90fc-4c81-8323-9f3a415a7c8e\") " Oct 29 22:12:59.484425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.484399 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca" pod "00329f2b-90fc-4c81-8323-9f3a415a7c8e" (UID: "00329f2b-90fc-4c81-8323-9f3a415a7c8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:59.484735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.484713 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:59.486320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.486300 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca" pod "00329f2b-90fc-4c81-8323-9f3a415a7c8e" (UID: "00329f2b-90fc-4c81-8323-9f3a415a7c8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:59.486410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.486391 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm" pod "00329f2b-90fc-4c81-8323-9f3a415a7c8e" (UID: "00329f2b-90fc-4c81-8323-9f3a415a7c8e"). InnerVolumeSpecName "kube-api-access-8j7gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:59.585737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.585700 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7gm\" (UniqueName: \"kubernetes.io/projected/00329f2b-90fc-4c81-8323-9f3a415a7c8e-kube-api-access-8j7gm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:12:59.585737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:12:59.585737 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00329f2b-90fc-4c81-8323-9f3a415a7c8e-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:00.221640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.221601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" event={"ID":"00329f2b-90fc-4c81-8323-9f3a415a7c8e","Type":"ContainerDied","Data":"8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767"} Oct 29 22:13:00.222067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.221662 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8819199700ff60f3ccc225c3956344a89fb9c339ca9d3605bd30f4e82d890767" Oct 29 22:13:00.222067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.221665 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4" Oct 29 22:13:00.847717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.847686 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nmzq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:13:00.847915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.847740 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 29 22:13:00.960758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:00.960734 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 22:13:01.033049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033004 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033090 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033168 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033212 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033273 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033310 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033377 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033377 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033449 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:01.033786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033503 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\") pod \"4783af05-3ee3-409a-a387-8c4a692c3d7e\" (UID: \"4783af05-3ee3-409a-a387-8c4a692c3d7e\") " Oct 29 22:13:01.033786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033640 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:01.033907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.033876 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:01.034396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.034282 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.034396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.034314 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4783af05-3ee3-409a-a387-8c4a692c3d7e-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.034396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.034329 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.034396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.034363 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.034396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.034377 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.036413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.036354 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "kube-api-access-j4phc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:01.036686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.036576 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:01.036686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.036599 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:01.036686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.036619 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo" pod "4783af05-3ee3-409a-a387-8c4a692c3d7e" (UID: "4783af05-3ee3-409a-a387-8c4a692c3d7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:13:01.135475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.135446 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.135475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.135471 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.135475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.135479 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4783af05-3ee3-409a-a387-8c4a692c3d7e-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.135698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.135489 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4phc\" (UniqueName: \"kubernetes.io/projected/4783af05-3ee3-409a-a387-8c4a692c3d7e-kube-api-access-j4phc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:01.232750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.232718 4543 generic.go:356] "Generic (PLEG): container finished" podID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerID="1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b" exitCode=0 Oct 29 22:13:01.233194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.232765 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" Oct 29 22:13:01.233194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.232797 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" event={"ID":"4783af05-3ee3-409a-a387-8c4a692c3d7e","Type":"ContainerDied","Data":"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b"} Oct 29 22:13:01.233194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.232859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9" event={"ID":"4783af05-3ee3-409a-a387-8c4a692c3d7e","Type":"ContainerDied","Data":"3887123cbfe3933120c790f42066cb96dbce64aa4dbcd5d050cc14096fe09872"} Oct 29 22:13:01.233194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.232887 4543 scope.go:122] "RemoveContainer" containerID="1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b" Oct 29 22:13:01.244801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.244783 4543 scope.go:122] "RemoveContainer" containerID="1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b" Oct 29 22:13:01.245093 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:01.245057 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b\": container with ID starting with 1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b not found: ID does not exist" containerID="1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b" Oct 29 22:13:01.245155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.245099 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b"} err="failed to get container status \"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b\": rpc error: code = NotFound desc = could not find container \"1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b\": container with ID starting with 1a68961425187a5d76e3b57a7bc1d4b58310491eaa7ea3e61fe5a4041766e14b not found: ID does not exist" Oct 29 22:13:01.296583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.296557 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 22:13:01.334098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:01.303811 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nmzq9"] Oct 29 22:13:02.386649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:02.386608 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" path="/var/lib/kubelet/pods/4783af05-3ee3-409a-a387-8c4a692c3d7e/volumes" Oct 29 22:13:04.422334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.422297 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:13:04.423331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423309 4543 cpu_manager.go:414] "containerMap: removing container" podUID="00329f2b-90fc-4c81-8323-9f3a415a7c8e" containerName="revoke-keys" Oct 29 22:13:04.423331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423330 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00329f2b-90fc-4c81-8323-9f3a415a7c8e" containerName="revoke-keys" Oct 29 22:13:04.423493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423347 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" Oct 29 22:13:04.423493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423355 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" Oct 29 22:13:04.423567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423510 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="00329f2b-90fc-4c81-8323-9f3a415a7c8e" containerName="revoke-keys" Oct 29 22:13:04.423567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.423523 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4783af05-3ee3-409a-a387-8c4a692c3d7e" containerName="istio-proxy" Oct 29 22:13:04.427948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.427926 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.460391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.460364 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-971004-openshift-default-dockercfg-kn96h\"" Oct 29 22:13:04.470337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.466315 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:13:04.480202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480168 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480277 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480332 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480378 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9q2\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.480790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.480608 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.481819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.481788 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.481942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.481846 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583412 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583454 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583569 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9q2\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583637 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583694 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583791 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583893 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.583996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.583916 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.584237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.584212 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.584311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.584248 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.584548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.584505 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.586037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.586014 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.586145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.586110 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.609218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.609195 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.609552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.609529 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9q2\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\") pod \"e2e-autores-971004-openshift-default-64cd89f865-rp4mj\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:04.745772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:04.745737 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:05.105711 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:05.105681 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc56a89_6948_401e_bf3f_b1be24f7c591.slice/crio-d5bcfff09a9ee4e8d737d2c8192a14918efd4a6144c193c03a41b17e28f2afbe WatchSource:0}: Error finding container d5bcfff09a9ee4e8d737d2c8192a14918efd4a6144c193c03a41b17e28f2afbe: Status 404 returned error can't find the container with id d5bcfff09a9ee4e8d737d2c8192a14918efd4a6144c193c03a41b17e28f2afbe Oct 29 22:13:05.106047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.106024 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:13:05.108265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.108234 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:05.108342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.108298 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:05.108342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.108328 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:05.254162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.254129 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:13:05.258779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.258761 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.276522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.276495 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-1e74c3\"" Oct 29 22:13:05.276522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.276526 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-1e74c3-dockercfg-zhdwl\"" Oct 29 22:13:05.282811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.282785 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:13:05.291241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.291211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerStarted","Data":"7c9e54eaf236ac6a6e3f297d83f26dde56fe9be33dc15b505a3642ea27ecaddd"} Oct 29 22:13:05.291364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.291269 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerStarted","Data":"d5bcfff09a9ee4e8d737d2c8192a14918efd4a6144c193c03a41b17e28f2afbe"} Oct 29 22:13:05.294286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.294261 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.294382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.294313 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbp9\" (UniqueName: \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.391149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.390594 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podStartSLOduration=1.390572545 podStartE2EDuration="1.390572545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:05.375780247 +0000 UTC m=+1847.117710932" watchObservedRunningTime="2026-10-29 22:13:05.390572545 +0000 UTC m=+1847.132503219" Oct 29 22:13:05.395101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.394129 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:13:05.408102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.399796 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.408102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.399877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbp9\" (UniqueName: \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.408102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.401693 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.417095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.415502 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.424097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.423662 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:13:05.585804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.585762 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbp9\" (UniqueName: \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\") pod \"payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.606053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.606016 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.606422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.606398 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbv6r\" (UniqueName: \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.707572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.707487 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.707731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.707600 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv6r\" (UniqueName: \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.708533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.708503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.734606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.734571 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv6r\" (UniqueName: \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\") pod \"payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:05.746160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.746136 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:05.876384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.876353 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:05.885714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:05.885687 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:06.233273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.233233 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:13:06.236014 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:06.235722 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53b8ee8_d5e3_41d8_a6d1_abb6cc0b69a8.slice/crio-ea420e3ee123df33667bb9714f7bb2512c0d7cded2e06b9b1cebc78f57fa2bc1 WatchSource:0}: Error finding container ea420e3ee123df33667bb9714f7bb2512c0d7cded2e06b9b1cebc78f57fa2bc1: Status 404 returned error can't find the container with id ea420e3ee123df33667bb9714f7bb2512c0d7cded2e06b9b1cebc78f57fa2bc1 Oct 29 22:13:06.258462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.258430 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:13:06.301808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.301779 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" event={"ID":"393e6e93-fcde-4371-adc3-b0a888f3041f","Type":"ContainerStarted","Data":"37ba81d2e77dbaefc68f1854a2985564a617a9ee31b826513b4418da3e8dd50f"} Oct 29 22:13:06.303275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.303247 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" event={"ID":"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8","Type":"ContainerStarted","Data":"ea420e3ee123df33667bb9714f7bb2512c0d7cded2e06b9b1cebc78f57fa2bc1"} Oct 29 22:13:06.752615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.752587 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:06.753465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.753443 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:13:06.791218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:06.791187 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:13:07.193388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.193321 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:13:07.193538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.193431 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:13:07.313639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.313611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" event={"ID":"393e6e93-fcde-4371-adc3-b0a888f3041f","Type":"ContainerStarted","Data":"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831"} Oct 29 22:13:07.313828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.313673 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:07.315484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.315457 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" event={"ID":"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8","Type":"ContainerStarted","Data":"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3"} Oct 29 22:13:07.315753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.315721 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:07.315879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.315763 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:07.316575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.316560 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:13:07.380923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.380868 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" podStartSLOduration=2.380855566 podStartE2EDuration="2.380855566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:07.376736294 +0000 UTC m=+1849.118666968" watchObservedRunningTime="2026-10-29 22:13:07.380855566 +0000 UTC m=+1849.122786254" Oct 29 22:13:07.473518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:07.473455 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" podStartSLOduration=2.473435388 podStartE2EDuration="2.473435388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:07.4668814 +0000 UTC m=+1849.208812074" watchObservedRunningTime="2026-10-29 22:13:07.473435388 +0000 UTC m=+1849.215366061" Oct 29 22:13:09.840217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.840184 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:13:09.843890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.843872 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:09.863972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.863948 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:13:09.871180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.871153 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:09.871395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.871372 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxzm\" (UniqueName: \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:09.871682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.871663 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-971004-dockercfg-fjfph\"" Oct 29 22:13:09.871682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.871663 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-971004\"" Oct 29 22:13:09.920213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.920189 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:13:09.924024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.924010 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:09.957386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.946872 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:13:09.984121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.982423 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxl69\" (UniqueName: \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:09.984121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.982582 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxzm\" (UniqueName: \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:09.984121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.982629 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:09.984121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:09.982838 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:10.076100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.067907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:10.132157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.131417 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxl69\" (UniqueName: \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:10.132157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.131573 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:10.193307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.132573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:10.193307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.192416 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxl69\" (UniqueName: \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\") pod \"payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:10.254353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.254317 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:10.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.350241 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxzm\" (UniqueName: \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\") pod \"payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:10.460195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.459859 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:10.648293 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:10.648264 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87774e3d_4618_4b45_9512_5296df11c1cd.slice/crio-49c29ef767d18d23f3444971aeb2bdf118e29022190c331e77e0546396c00f28 WatchSource:0}: Error finding container 49c29ef767d18d23f3444971aeb2bdf118e29022190c331e77e0546396c00f28: Status 404 returned error can't find the container with id 49c29ef767d18d23f3444971aeb2bdf118e29022190c331e77e0546396c00f28 Oct 29 22:13:10.648420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.648400 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:13:10.665233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.665213 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n"] Oct 29 22:13:10.671574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.671552 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7721b3-r8k7n"] Oct 29 22:13:10.807228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:10.807195 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:13:10.807883 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:10.807857 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719ea2db_6ed7_4bd0_88ba_d2bc5e3be0c9.slice/crio-718157c460e041a27a72f16bcb6960ffa8b50c904e6d057e356059f6a59d313d WatchSource:0}: Error finding container 718157c460e041a27a72f16bcb6960ffa8b50c904e6d057e356059f6a59d313d: Status 404 returned error can't find the container with id 718157c460e041a27a72f16bcb6960ffa8b50c904e6d057e356059f6a59d313d Oct 29 22:13:11.375570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.375535 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" event={"ID":"87774e3d-4618-4b45-9512-5296df11c1cd","Type":"ContainerStarted","Data":"fc7d35cd87168dca6471938c4ce0477d52049d20265dfbd7c99f2a0f1eddad1a"} Oct 29 22:13:11.376008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.375588 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" event={"ID":"87774e3d-4618-4b45-9512-5296df11c1cd","Type":"ContainerStarted","Data":"49c29ef767d18d23f3444971aeb2bdf118e29022190c331e77e0546396c00f28"} Oct 29 22:13:11.376008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.375614 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:11.377467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.377433 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" event={"ID":"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9","Type":"ContainerStarted","Data":"4c5c955f3cf9a255e49d467ba50ae3c09acce2967a7cb59ad33d1df4d688fd3e"} Oct 29 22:13:11.377586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.377475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" event={"ID":"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9","Type":"ContainerStarted","Data":"718157c460e041a27a72f16bcb6960ffa8b50c904e6d057e356059f6a59d313d"} Oct 29 22:13:11.377586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.377495 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:11.421035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.420991 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" podStartSLOduration=2.420977741 podStartE2EDuration="2.420977741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:11.418808448 +0000 UTC m=+1853.160739122" watchObservedRunningTime="2026-10-29 22:13:11.420977741 +0000 UTC m=+1853.162908460" Oct 29 22:13:11.464255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.464205 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" podStartSLOduration=2.464192943 podStartE2EDuration="2.464192943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:11.462789274 +0000 UTC m=+1853.204719946" watchObservedRunningTime="2026-10-29 22:13:11.464192943 +0000 UTC m=+1853.206123615" Oct 29 22:13:11.754181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.754124 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" podUID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" containerName="payload-processing" containerID="cri-o://16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911" gracePeriod=25 Oct 29 22:13:11.791453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:11.791422 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" podUID="9d863c8d-b4e7-4fe3-a461-29e144aff444" containerName="payload-pre-processing" containerID="cri-o://26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5" gracePeriod=25 Oct 29 22:13:12.075446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.075410 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:13:12.097492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.097466 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:13:12.170345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.170309 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\") pod \"9d863c8d-b4e7-4fe3-a461-29e144aff444\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " Oct 29 22:13:12.170659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.170634 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\") pod \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " Oct 29 22:13:12.170740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.170720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\" (UniqueName: \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\") pod \"9d863c8d-b4e7-4fe3-a461-29e144aff444\" (UID: \"9d863c8d-b4e7-4fe3-a461-29e144aff444\") " Oct 29 22:13:12.170784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.170748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\" (UniqueName: \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\") pod \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\" (UID: \"bd11e93a-0e42-4362-b1f1-3284b29d1e17\") " Oct 29 22:13:12.170897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.170866 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume" pod "9d863c8d-b4e7-4fe3-a461-29e144aff444" (UID: "9d863c8d-b4e7-4fe3-a461-29e144aff444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:12.171346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.171321 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume" pod "bd11e93a-0e42-4362-b1f1-3284b29d1e17" (UID: "bd11e93a-0e42-4362-b1f1-3284b29d1e17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:12.171462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.171441 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d863c8d-b4e7-4fe3-a461-29e144aff444-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:12.171505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.171473 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd11e93a-0e42-4362-b1f1-3284b29d1e17-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:12.173597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.173568 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62" pod "9d863c8d-b4e7-4fe3-a461-29e144aff444" (UID: "9d863c8d-b4e7-4fe3-a461-29e144aff444"). InnerVolumeSpecName "kube-api-access-zqk62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:12.173982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.173960 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm" pod "bd11e93a-0e42-4362-b1f1-3284b29d1e17" (UID: "bd11e93a-0e42-4362-b1f1-3284b29d1e17"). InnerVolumeSpecName "kube-api-access-mp5dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:12.282143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.273421 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqk62\" (UniqueName: \"kubernetes.io/projected/9d863c8d-b4e7-4fe3-a461-29e144aff444-kube-api-access-zqk62\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:12.282143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.273477 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp5dm\" (UniqueName: \"kubernetes.io/projected/bd11e93a-0e42-4362-b1f1-3284b29d1e17-kube-api-access-mp5dm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:12.386727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.386698 4543 generic.go:356] "Generic (PLEG): container finished" podID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" containerID="16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911" exitCode=0 Oct 29 22:13:12.387167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.386797 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0b2ac8-057d-47f4-823a-8575f7ce4cbe" path="/var/lib/kubelet/pods/ee0b2ac8-057d-47f4-823a-8575f7ce4cbe/volumes" Oct 29 22:13:12.387167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.386832 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" Oct 29 22:13:12.387528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.387501 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" event={"ID":"bd11e93a-0e42-4362-b1f1-3284b29d1e17","Type":"ContainerDied","Data":"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911"} Oct 29 22:13:12.387665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.387545 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw" event={"ID":"bd11e93a-0e42-4362-b1f1-3284b29d1e17","Type":"ContainerDied","Data":"4bd177e3b6d57f31bdc27dc55543c853dbb6bbed92cd71fe60e2220a08a992d7"} Oct 29 22:13:12.387665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.387571 4543 scope.go:122] "RemoveContainer" containerID="16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911" Oct 29 22:13:12.388750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.388553 4543 generic.go:356] "Generic (PLEG): container finished" podID="9d863c8d-b4e7-4fe3-a461-29e144aff444" containerID="26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5" exitCode=0 Oct 29 22:13:12.388750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.388576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" event={"ID":"9d863c8d-b4e7-4fe3-a461-29e144aff444","Type":"ContainerDied","Data":"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5"} Oct 29 22:13:12.388750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.388611 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" Oct 29 22:13:12.388750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.388623 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw" event={"ID":"9d863c8d-b4e7-4fe3-a461-29e144aff444","Type":"ContainerDied","Data":"9df19a8ad465199f43a9dc8c270b619a433c1bede77de31e7ab8372589e255e5"} Oct 29 22:13:12.401794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.401776 4543 scope.go:122] "RemoveContainer" containerID="16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911" Oct 29 22:13:12.402110 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:12.402051 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911\": container with ID starting with 16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911 not found: ID does not exist" containerID="16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911" Oct 29 22:13:12.402214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.402101 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911"} err="failed to get container status \"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911\": rpc error: code = NotFound desc = could not find container \"16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911\": container with ID starting with 16564e37c8e993161aeab4f8d640ff096833d050f3964d9d6fd3a32ad12a1911 not found: ID does not exist" Oct 29 22:13:12.402214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.402125 4543 scope.go:122] "RemoveContainer" containerID="26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5" Oct 29 22:13:12.412433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.412418 4543 scope.go:122] "RemoveContainer" containerID="26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5" Oct 29 22:13:12.412670 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:12.412652 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5\": container with ID starting with 26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5 not found: ID does not exist" containerID="26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5" Oct 29 22:13:12.412712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.412677 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5"} err="failed to get container status \"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5\": rpc error: code = NotFound desc = could not find container \"26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5\": container with ID starting with 26c6d14ef6441326ee60f9076c439a24bbbcbf906ad34f003b9854b79d1f97f5 not found: ID does not exist" Oct 29 22:13:12.464533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.464493 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:13:12.470301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.470276 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b0939-84498c9f9f-4p9vw"] Oct 29 22:13:12.508246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.508226 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:13:12.513276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:12.513257 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b0939-6788bdd8f8-9hbbw"] Oct 29 22:13:14.387596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:14.387560 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d863c8d-b4e7-4fe3-a461-29e144aff444" path="/var/lib/kubelet/pods/9d863c8d-b4e7-4fe3-a461-29e144aff444/volumes" Oct 29 22:13:14.387988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:14.387973 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" path="/var/lib/kubelet/pods/bd11e93a-0e42-4362-b1f1-3284b29d1e17/volumes" Oct 29 22:13:18.357281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.357250 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:13:18.359194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.359172 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:13:18.721464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.721425 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b"] Oct 29 22:13:18.722152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722133 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9d863c8d-b4e7-4fe3-a461-29e144aff444" containerName="payload-pre-processing" Oct 29 22:13:18.722214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722154 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d863c8d-b4e7-4fe3-a461-29e144aff444" containerName="payload-pre-processing" Oct 29 22:13:18.722214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722183 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" containerName="payload-processing" Oct 29 22:13:18.722214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722189 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" containerName="payload-processing" Oct 29 22:13:18.722339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722329 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d863c8d-b4e7-4fe3-a461-29e144aff444" containerName="payload-pre-processing" Oct 29 22:13:18.722375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.722345 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd11e93a-0e42-4362-b1f1-3284b29d1e17" containerName="payload-processing" Oct 29 22:13:18.727843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.727827 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.744391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.744367 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b"] Oct 29 22:13:18.752122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.752097 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:13:18.752244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.752172 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:13:18.783612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.783589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.783724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.783654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxg2\" (UniqueName: \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.783848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.783828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.799401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.799381 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:13:18.799500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.799427 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:13:18.885824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.885801 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.885913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.885862 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxg2\" (UniqueName: \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.885969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.885939 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.886637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.886618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.888422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.888404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:18.913383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:18.913355 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxg2\" (UniqueName: \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\") pod \"maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:19.042194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:19.042145 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:19.354157 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:19.354124 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fdc7fc_d621_4447_bab1_092df02d7a84.slice/crio-62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2 WatchSource:0}: Error finding container 62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2: Status 404 returned error can't find the container with id 62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2 Oct 29 22:13:19.354665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:19.354637 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b"] Oct 29 22:13:19.465137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:19.465112 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" event={"ID":"77fdc7fc-d621-4447-bab1-092df02d7a84","Type":"ContainerStarted","Data":"62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2"} Oct 29 22:13:20.474987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:20.474949 4543 generic.go:356] "Generic (PLEG): container finished" podID="77fdc7fc-d621-4447-bab1-092df02d7a84" containerID="5bb362c913cae881413e93cce2d333f9127f8f4a693a601c6ccf8e3cf63d2461" exitCode=0 Oct 29 22:13:20.475454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:20.475029 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" event={"ID":"77fdc7fc-d621-4447-bab1-092df02d7a84","Type":"ContainerDied","Data":"5bb362c913cae881413e93cce2d333f9127f8f4a693a601c6ccf8e3cf63d2461"} Oct 29 22:13:20.722034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:20.722005 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4"] Oct 29 22:13:20.727427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:20.727404 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b0939-76tc4"] Oct 29 22:13:21.641624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.641601 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:21.747442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.747347 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\" (UniqueName: \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\") pod \"77fdc7fc-d621-4447-bab1-092df02d7a84\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " Oct 29 22:13:21.747607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.747519 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\") pod \"77fdc7fc-d621-4447-bab1-092df02d7a84\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " Oct 29 22:13:21.747607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.747593 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\") pod \"77fdc7fc-d621-4447-bab1-092df02d7a84\" (UID: \"77fdc7fc-d621-4447-bab1-092df02d7a84\") " Oct 29 22:13:21.748440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.748408 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca" pod "77fdc7fc-d621-4447-bab1-092df02d7a84" (UID: "77fdc7fc-d621-4447-bab1-092df02d7a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:21.750770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.750739 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2" pod "77fdc7fc-d621-4447-bab1-092df02d7a84" (UID: "77fdc7fc-d621-4447-bab1-092df02d7a84"). InnerVolumeSpecName "kube-api-access-4vxg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:21.751125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.751065 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca" pod "77fdc7fc-d621-4447-bab1-092df02d7a84" (UID: "77fdc7fc-d621-4447-bab1-092df02d7a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:21.760181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.760157 4543 scope.go:122] "RemoveContainer" containerID="f82747d89f7a57b586730726bac1613b83946453394cedd234e6c3a62f165757" Oct 29 22:13:21.791913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.791870 4543 scope.go:122] "RemoveContainer" containerID="66510eee1c3a8b0bdd5893e968b53ee7df207b103c66605770a6b202b45ac972" Oct 29 22:13:21.850488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.850466 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxg2\" (UniqueName: \"kubernetes.io/projected/77fdc7fc-d621-4447-bab1-092df02d7a84-kube-api-access-4vxg2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:21.850580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.850492 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:21.850580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:21.850502 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc7fc-d621-4447-bab1-092df02d7a84-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:22.399690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.399647 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00329f2b-90fc-4c81-8323-9f3a415a7c8e" path="/var/lib/kubelet/pods/00329f2b-90fc-4c81-8323-9f3a415a7c8e/volumes" Oct 29 22:13:22.400777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.400750 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:13:22.400952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.400865 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:13:22.504580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.504551 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" Oct 29 22:13:22.504580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.504565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b" event={"ID":"77fdc7fc-d621-4447-bab1-092df02d7a84","Type":"ContainerDied","Data":"62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2"} Oct 29 22:13:22.504805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:22.504632 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fd2137a510d96e50ce6e012e52a511e96dfbc4a2d0a510faaa41b2a097f0f2" Oct 29 22:13:27.982869 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:27.982833 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:13:28.698737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:28.698659 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:13:28.699189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:28.699161 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" containerID="cri-o://ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3" gracePeriod=30 Oct 29 22:13:30.595267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:30.595223 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:13:30.595793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:30.595317 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:13:32.633771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.633734 4543 generic.go:356] "Generic (PLEG): container finished" podID="db05227c-5018-4052-a277-4864230a5a65" containerID="1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8" exitCode=0 Oct 29 22:13:32.634424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.633813 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerDied","Data":"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8"} Oct 29 22:13:32.634424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.634340 4543 scope.go:122] "RemoveContainer" containerID="1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8" Oct 29 22:13:32.636323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.636289 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:32.636453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.636380 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:32.636453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:32.636431 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:33.664617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:33.664576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerStarted","Data":"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435"} Oct 29 22:13:34.070375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.070350 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:13:34.223930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.223901 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224010 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224125 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224176 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224208 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224218 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:34.224337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224299 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224391 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224433 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224435 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:34.224561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:34.224561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224552 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\") pod \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\" (UID: \"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5\") " Oct 29 22:13:34.224702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224678 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:34.224764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.224728 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:34.225390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.225364 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.225390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.225388 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.225585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.225397 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.225585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.225406 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.225585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.225414 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.226838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.226809 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:13:34.226838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.226832 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:13:34.227139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.227117 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "kube-api-access-mr4r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:34.227341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.227322 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token" pod "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" (UID: "07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:34.327059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.326990 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr4r8\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-kube-api-access-mr4r8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.327059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.327015 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.327059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.327025 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.327059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.327034 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:34.468225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.468193 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:13:34.468371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.468326 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 29 22:13:34.468415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.468371 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 29 22:13:34.674886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.674811 4543 generic.go:356] "Generic (PLEG): container finished" podID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerID="ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3" exitCode=0 Oct 29 22:13:34.675318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.674881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerDied","Data":"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3"} Oct 29 22:13:34.675318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.674925 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" Oct 29 22:13:34.675318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.674931 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5" event={"ID":"07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5","Type":"ContainerDied","Data":"661ec643f5dc6fd49d35e71ca092790d218f3d9faa173f2263eeaf1271af0f53"} Oct 29 22:13:34.675318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.674952 4543 scope.go:122] "RemoveContainer" containerID="ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3" Oct 29 22:13:34.686400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.686381 4543 scope.go:122] "RemoveContainer" containerID="9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b" Oct 29 22:13:34.697209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.697089 4543 scope.go:122] "RemoveContainer" containerID="ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3" Oct 29 22:13:34.697406 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:34.697385 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3\": container with ID starting with ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3 not found: ID does not exist" containerID="ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3" Oct 29 22:13:34.697480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.697414 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3"} err="failed to get container status \"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3\": rpc error: code = NotFound desc = could not find container \"ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3\": container with ID starting with ad22481eb00d020822d3a116700f673419363c8fd2db60a9053eb5847de57ae3 not found: ID does not exist" Oct 29 22:13:34.697480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.697429 4543 scope.go:122] "RemoveContainer" containerID="9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b" Oct 29 22:13:34.697689 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:34.697671 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b\": container with ID starting with 9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b not found: ID does not exist" containerID="9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b" Oct 29 22:13:34.697737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.697696 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b"} err="failed to get container status \"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b\": rpc error: code = NotFound desc = could not find container \"9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b\": container with ID starting with 9a73b8c26ca9ddf931df194a5254970b2823ace1074feefc7d6e614d5d7f639b not found: ID does not exist" Oct 29 22:13:34.762014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.761810 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:13:34.766820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:34.766382 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b0939-openshift-default-f5ff767cc-wbxh5"] Oct 29 22:13:35.468098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:35.468033 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 29 22:13:35.468274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:35.468113 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 29 22:13:36.387410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.387380 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" path="/var/lib/kubelet/pods/07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5/volumes" Oct 29 22:13:36.467722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.467690 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 29 22:13:36.467893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.467745 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 29 22:13:36.763349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.763314 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:13:36.764652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.764621 4543 cpu_manager.go:414] "containerMap: removing container" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.779159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779116 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.779311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779187 4543 cpu_manager.go:414] "containerMap: removing container" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.779311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779196 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.779311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779269 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77fdc7fc-d621-4447-bab1-092df02d7a84" containerName="revoke-keys" Oct 29 22:13:36.779311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779276 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fdc7fc-d621-4447-bab1-092df02d7a84" containerName="revoke-keys" Oct 29 22:13:36.779535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779489 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.779535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.779515 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fdc7fc-d621-4447-bab1-092df02d7a84" containerName="revoke-keys" Oct 29 22:13:36.789106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.789063 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.790444 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6b37a-36f0-4d25-8c5e-0f7dd13a99f5" containerName="istio-proxy" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.789209 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.792850 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.792886 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqw5\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.792991 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793102 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793156 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793201 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793225 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.797106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.793348 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.810170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.809628 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:13:36.810170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.809671 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:13:36.810170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.809759 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.894055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894246 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.894368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894357 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.894444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894434 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894545 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894611 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.894699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894691 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.894766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894758 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894895 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894936 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.894968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.894941 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.895105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.895155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895142 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895183 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895196 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895324 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5nk\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.895548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895564 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqw5\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.895636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.895697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.895637 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:36.896070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.896052 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.896904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.896887 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.898296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.898279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.920719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.920700 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:36.921304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:36.921283 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqw5\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:37.001356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.001551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001496 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.001551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001533 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.001692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001644 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5nk\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.001692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001680 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.001809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001717 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.001977 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002043 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002142 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002189 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002250 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002288 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002637 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.002821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.002801 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.004596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.004568 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.004729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.004708 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.054309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.054178 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5nk\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.055354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.055331 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.143933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.143278 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:37.189031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.189009 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:37.468440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.468404 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 29 22:13:37.468825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.468461 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 29 22:13:37.490338 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:37.490312 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178fc4c0_85ee_4f87_932d_8a51e96cb0cf.slice/crio-61907b1b031bf7a2941313914c571aaeacf374afde1ee709442d33ac5be2fa18 WatchSource:0}: Error finding container 61907b1b031bf7a2941313914c571aaeacf374afde1ee709442d33ac5be2fa18: Status 404 returned error can't find the container with id 61907b1b031bf7a2941313914c571aaeacf374afde1ee709442d33ac5be2fa18 Oct 29 22:13:37.490882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.490813 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:13:37.493207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.493172 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.493327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.493248 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.493327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.493291 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.584203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.570759 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:13:37.588859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.588822 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.589021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.589006 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.589156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.589124 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:37.709096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.708417 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" event={"ID":"974058c5-b257-479b-a8e9-1c37b7641c98","Type":"ContainerStarted","Data":"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33"} Oct 29 22:13:37.709096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.708480 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" event={"ID":"974058c5-b257-479b-a8e9-1c37b7641c98","Type":"ContainerStarted","Data":"fa4369e5713d3f36daf9d31c79341d303572410198c8bd22ba70445c5a8d10b2"} Oct 29 22:13:37.711526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.710773 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" event={"ID":"178fc4c0-85ee-4f87-932d-8a51e96cb0cf","Type":"ContainerStarted","Data":"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e"} Oct 29 22:13:37.711526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.710826 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" event={"ID":"178fc4c0-85ee-4f87-932d-8a51e96cb0cf","Type":"ContainerStarted","Data":"61907b1b031bf7a2941313914c571aaeacf374afde1ee709442d33ac5be2fa18"} Oct 29 22:13:37.771300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.769676 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podStartSLOduration=1.769657387 podStartE2EDuration="1.769657387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:37.76688582 +0000 UTC m=+1879.508816495" watchObservedRunningTime="2026-10-29 22:13:37.769657387 +0000 UTC m=+1879.511588068" Oct 29 22:13:37.857617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:37.857503 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podStartSLOduration=1.8574842230000002 podStartE2EDuration="1.857484223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:37.815536331 +0000 UTC m=+1879.557467008" watchObservedRunningTime="2026-10-29 22:13:37.857484223 +0000 UTC m=+1879.599414895" Oct 29 22:13:38.149521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.149436 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:38.156214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.156180 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:38.156434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.156408 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:38.168529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.168502 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2"] Oct 29 22:13:38.173488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.173468 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.189705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.189679 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:38.191004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.190979 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:38.191143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.191029 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:38.193161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.193127 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:13:38.193391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.193375 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:13:38.193484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.193470 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:13:38.193541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.193530 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:13:38.193887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.193870 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2"] Oct 29 22:13:38.410113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.409881 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.410348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.410305 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.410478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.410455 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sws64\" (UniqueName: \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.483099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.482473 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 29 22:13:38.483099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.482538 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 29 22:13:38.512366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.512331 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.512560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.512426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.512560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.512479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sws64\" (UniqueName: \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.513181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.513161 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.515041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.515019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.554099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.553164 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sws64\" (UniqueName: \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\") pod \"maas-api-revoke-sub-8e03a25d-k76j2\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:38.794649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:38.794607 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:39.177609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.177517 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:39.177875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.177843 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:39.192276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.191190 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:39.192532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.192503 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:39.297436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.297398 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2"] Oct 29 22:13:39.498002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.497973 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:13:39.499743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.499724 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:13:39.502050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.501803 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:13:39.802948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:39.800538 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" event={"ID":"6efbb881-d26c-473b-983b-5e208e00c104","Type":"ContainerStarted","Data":"f483383637a17f38156992b721c05dbf6f3fb7e9e1b49e0d6b0d816383422890"} Oct 29 22:13:40.163161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.157865 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:40.163161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.157929 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:40.189609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.189572 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:40.189771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.189622 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:40.813396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.813361 4543 generic.go:356] "Generic (PLEG): container finished" podID="6efbb881-d26c-473b-983b-5e208e00c104" containerID="9e4c1690601ad2a8ed32cd0d7d36fd7b02f8adb7290a91839634c3f2018127ad" exitCode=0 Oct 29 22:13:40.814534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:40.814504 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" event={"ID":"6efbb881-d26c-473b-983b-5e208e00c104","Type":"ContainerDied","Data":"9e4c1690601ad2a8ed32cd0d7d36fd7b02f8adb7290a91839634c3f2018127ad"} Oct 29 22:13:41.152102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:41.149445 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:41.152102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:41.149549 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:41.190240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:41.190208 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:41.190380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:41.190253 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:42.190174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.189463 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:42.190174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.189522 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:42.190174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.189937 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:42.190174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.189979 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:42.226237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.226213 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:42.320453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.320418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\") pod \"6efbb881-d26c-473b-983b-5e208e00c104\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " Oct 29 22:13:42.320622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.320597 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\") pod \"6efbb881-d26c-473b-983b-5e208e00c104\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " Oct 29 22:13:42.320685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.320648 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\" (UniqueName: \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\") pod \"6efbb881-d26c-473b-983b-5e208e00c104\" (UID: \"6efbb881-d26c-473b-983b-5e208e00c104\") " Oct 29 22:13:42.321155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.321126 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca" pod "6efbb881-d26c-473b-983b-5e208e00c104" (UID: "6efbb881-d26c-473b-983b-5e208e00c104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:42.321535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.321468 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:42.324618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.323200 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64" pod "6efbb881-d26c-473b-983b-5e208e00c104" (UID: "6efbb881-d26c-473b-983b-5e208e00c104"). InnerVolumeSpecName "kube-api-access-sws64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:42.324618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.323564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca" pod "6efbb881-d26c-473b-983b-5e208e00c104" (UID: "6efbb881-d26c-473b-983b-5e208e00c104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:42.422576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.422548 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efbb881-d26c-473b-983b-5e208e00c104-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:42.422576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.422573 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sws64\" (UniqueName: \"kubernetes.io/projected/6efbb881-d26c-473b-983b-5e208e00c104-kube-api-access-sws64\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:42.881004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.880966 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" event={"ID":"6efbb881-d26c-473b-983b-5e208e00c104","Type":"ContainerDied","Data":"f483383637a17f38156992b721c05dbf6f3fb7e9e1b49e0d6b0d816383422890"} Oct 29 22:13:42.881395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.881357 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f483383637a17f38156992b721c05dbf6f3fb7e9e1b49e0d6b0d816383422890" Oct 29 22:13:42.881531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:42.881180 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2" Oct 29 22:13:43.164152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:43.163471 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 29 22:13:43.164152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:43.163528 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 29 22:13:43.189554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:43.189521 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 29 22:13:43.189724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:43.189576 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 29 22:13:44.151754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.151726 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:44.152210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.152187 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:44.153048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.153030 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:13:44.193637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.193613 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:44.193900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.193881 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:44.194601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:44.194578 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:13:48.521648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.521617 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh"] Oct 29 22:13:48.522358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.522340 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6efbb881-d26c-473b-983b-5e208e00c104" containerName="revoke-keys" Oct 29 22:13:48.522408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.522360 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efbb881-d26c-473b-983b-5e208e00c104" containerName="revoke-keys" Oct 29 22:13:48.522484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.522475 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efbb881-d26c-473b-983b-5e208e00c104" containerName="revoke-keys" Oct 29 22:13:48.525316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.525299 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.545129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.545107 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh"] Oct 29 22:13:48.553176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.553151 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:13:48.553290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.553158 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:13:48.553290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.553233 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:13:48.555029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.555011 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:13:48.579353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.579330 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.579446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.579416 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzrk\" (UniqueName: \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.579576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.579560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.680572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.680544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzrk\" (UniqueName: \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.680699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.680681 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.680841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.680824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.681474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.681452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.683200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.683182 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.706103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.706065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzrk\" (UniqueName: \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\") pod \"maas-api-revoke-sub-59a3b57f-6mhgh\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:48.839753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:48.839696 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:49.196977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:49.196945 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh"] Oct 29 22:13:49.197450 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:49.197427 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eed4bef_76c4_41d7_9fbc_7e69acc82387.slice/crio-692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9 WatchSource:0}: Error finding container 692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9: Status 404 returned error can't find the container with id 692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9 Oct 29 22:13:49.957840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:49.957766 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" event={"ID":"5eed4bef-76c4-41d7-9fbc-7e69acc82387","Type":"ContainerStarted","Data":"56bf78a74874b42192d90cd3082ffca63bdf17712df92d80bd0bcc4e9f2fa0b5"} Oct 29 22:13:49.957840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:49.957817 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" event={"ID":"5eed4bef-76c4-41d7-9fbc-7e69acc82387","Type":"ContainerStarted","Data":"692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9"} Oct 29 22:13:50.002773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:50.002726 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" podStartSLOduration=2.002714321 podStartE2EDuration="2.002714321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:50.000301726 +0000 UTC m=+1891.742232399" watchObservedRunningTime="2026-10-29 22:13:50.002714321 +0000 UTC m=+1891.744645011" Oct 29 22:13:50.970984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:50.970928 4543 generic.go:356] "Generic (PLEG): container finished" podID="5eed4bef-76c4-41d7-9fbc-7e69acc82387" containerID="56bf78a74874b42192d90cd3082ffca63bdf17712df92d80bd0bcc4e9f2fa0b5" exitCode=0 Oct 29 22:13:50.971704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:50.971312 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" event={"ID":"5eed4bef-76c4-41d7-9fbc-7e69acc82387","Type":"ContainerDied","Data":"56bf78a74874b42192d90cd3082ffca63bdf17712df92d80bd0bcc4e9f2fa0b5"} Oct 29 22:13:51.641707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:51.641674 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b"] Oct 29 22:13:51.654125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:51.654093 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4c5549-65d9b"] Oct 29 22:13:52.153009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.152983 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:52.251790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.251721 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\") pod \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " Oct 29 22:13:52.251931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.251842 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\") pod \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " Oct 29 22:13:52.251995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.251982 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\" (UniqueName: \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\") pod \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\" (UID: \"5eed4bef-76c4-41d7-9fbc-7e69acc82387\") " Oct 29 22:13:52.252270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.252236 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca" pod "5eed4bef-76c4-41d7-9fbc-7e69acc82387" (UID: "5eed4bef-76c4-41d7-9fbc-7e69acc82387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:52.252413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.252395 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:52.253926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.253902 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca" pod "5eed4bef-76c4-41d7-9fbc-7e69acc82387" (UID: "5eed4bef-76c4-41d7-9fbc-7e69acc82387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:52.254259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.254236 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk" pod "5eed4bef-76c4-41d7-9fbc-7e69acc82387" (UID: "5eed4bef-76c4-41d7-9fbc-7e69acc82387"). InnerVolumeSpecName "kube-api-access-gkzrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:52.354030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.354006 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzrk\" (UniqueName: \"kubernetes.io/projected/5eed4bef-76c4-41d7-9fbc-7e69acc82387-kube-api-access-gkzrk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:52.354130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.354032 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eed4bef-76c4-41d7-9fbc-7e69acc82387-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:13:52.388192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.388165 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fdc7fc-d621-4447-bab1-092df02d7a84" path="/var/lib/kubelet/pods/77fdc7fc-d621-4447-bab1-092df02d7a84/volumes" Oct 29 22:13:52.990140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.990111 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" Oct 29 22:13:52.990316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.990112 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh" event={"ID":"5eed4bef-76c4-41d7-9fbc-7e69acc82387","Type":"ContainerDied","Data":"692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9"} Oct 29 22:13:52.990316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:52.990252 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692c2a0952a560cb362bd5074ecb9a44eadc59958bc6834cc0fb27efcc705fb9" Oct 29 22:13:56.304067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.288136 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:13:56.304067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.289280 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5eed4bef-76c4-41d7-9fbc-7e69acc82387" containerName="revoke-keys" Oct 29 22:13:56.304067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.289312 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eed4bef-76c4-41d7-9fbc-7e69acc82387" containerName="revoke-keys" Oct 29 22:13:56.304067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.289502 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eed4bef-76c4-41d7-9fbc-7e69acc82387" containerName="revoke-keys" Oct 29 22:13:56.304067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.293619 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.322904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.322875 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:13:56.328427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.328400 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-675bf4-openshift-default-dockercfg-pj8l2\"" Oct 29 22:13:56.442100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.422769 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.442507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.442478 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.442659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.442640 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.442839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.442821 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcq9\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.442979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.442963 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.449200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.443124 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.449486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.449461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.463096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.459828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.463096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.459940 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.562330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.562232 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.563484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.563360 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.563722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.563704 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.564219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.564197 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.564670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.564653 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.565104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.564601 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.565216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.565025 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.565408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.565391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcq9\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.568240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.568219 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.568773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.568756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.569635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.569614 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.570251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.570231 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.570799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.569520 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.570923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.569996 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.596120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.589907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.596120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.595667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.604414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.603670 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.609233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.605403 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcq9\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\") pod \"e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.617375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.616472 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:56.979829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.979804 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:13:56.980306 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:13:56.980279 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f04497_2263_427b_8521_d79e20303f98.slice/crio-fc3f72a34f17e6aaef2a9b297be84c5d366e9ff77cd5bd3d41f5aa2cae76eee4 WatchSource:0}: Error finding container fc3f72a34f17e6aaef2a9b297be84c5d366e9ff77cd5bd3d41f5aa2cae76eee4: Status 404 returned error can't find the container with id fc3f72a34f17e6aaef2a9b297be84c5d366e9ff77cd5bd3d41f5aa2cae76eee4 Oct 29 22:13:56.982940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.982906 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:56.983021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.982977 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:56.983021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:56.983006 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:13:57.027644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:57.027616 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerStarted","Data":"fc3f72a34f17e6aaef2a9b297be84c5d366e9ff77cd5bd3d41f5aa2cae76eee4"} Oct 29 22:13:57.962536 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:13:57.962493 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:13:58.038455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:58.038424 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerStarted","Data":"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e"} Oct 29 22:13:58.109710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:58.109653 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podStartSLOduration=2.109638705 podStartE2EDuration="2.109638705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:58.106003911 +0000 UTC m=+1899.847934585" watchObservedRunningTime="2026-10-29 22:13:58.109638705 +0000 UTC m=+1899.851569379" Oct 29 22:13:58.618603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:58.618572 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:58.624188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:58.624166 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:59.048388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:59.048358 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:13:59.049233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:13:59.049211 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:14:02.666553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.666522 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:14:02.671049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.671032 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.694698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.694672 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:14:02.701468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.701445 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-675bf4\"" Oct 29 22:14:02.701600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.701482 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-675bf4-dockercfg-xtwsk\"" Oct 29 22:14:02.730461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.730441 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:14:02.734999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.734979 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:02.908115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.877856 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbhj\" (UniqueName: \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.908115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.877908 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:02.908115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.878130 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvkk\" (UniqueName: \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:02.908115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.878200 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.934154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.933407 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:14:02.995150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.995105 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvkk\" (UniqueName: \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:02.995368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.995201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.995446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.995396 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbhj\" (UniqueName: \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.995446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.995423 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:02.997234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.997047 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:02.997234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:02.997162 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:03.021868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:03.021836 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvkk\" (UniqueName: \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\") pod \"payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:03.024033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:03.023377 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbhj\" (UniqueName: \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\") pod \"payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:03.263126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:03.263094 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:03.286682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:03.286654 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:03.945282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:03.940837 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:14:04.554257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:04.540425 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" event={"ID":"16b748bc-aa36-4543-b56a-519ccdc8e61f","Type":"ContainerStarted","Data":"4eef7937ed8f0ea727a88c31d7d3d769f3733145984620797bdbcec17e3778be"} Oct 29 22:14:04.554257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:04.540495 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:14:05.498670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.498634 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb"] Oct 29 22:14:05.521338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.521302 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" event={"ID":"16b748bc-aa36-4543-b56a-519ccdc8e61f","Type":"ContainerStarted","Data":"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b"} Oct 29 22:14:05.522748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.521548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" event={"ID":"4b387aab-8504-45c5-8fb8-da1667bc3d7b","Type":"ContainerStarted","Data":"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3"} Oct 29 22:14:05.522748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.521580 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" event={"ID":"4b387aab-8504-45c5-8fb8-da1667bc3d7b","Type":"ContainerStarted","Data":"b24e4fe37bf430bdefdc9c3e6eed17fb700ca05c1e6b79f4eb30f49e8c702b90"} Oct 29 22:14:05.522748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.521606 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:05.522748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.521686 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.526420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.524520 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:05.557534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.555373 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:14:05.557534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.555503 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:14:05.557534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.555647 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:14:05.606566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.606537 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb"] Oct 29 22:14:05.626723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.626666 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" podStartSLOduration=3.626646688 podStartE2EDuration="3.626646688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:05.568475009 +0000 UTC m=+1907.310405684" watchObservedRunningTime="2026-10-29 22:14:05.626646688 +0000 UTC m=+1907.368577353" Oct 29 22:14:05.719098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.699059 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27cj\" (UniqueName: \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.719098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.699399 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.719098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.699454 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.719098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.705413 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:14:05.929213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.901545 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" podStartSLOduration=3.901521702 podStartE2EDuration="3.901521702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:05.70561202 +0000 UTC m=+1907.447542694" watchObservedRunningTime="2026-10-29 22:14:05.901521702 +0000 UTC m=+1907.643452375" Oct 29 22:14:05.929213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.918466 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27cj\" (UniqueName: \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.929213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.918779 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.929213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.918837 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.929213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.919757 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.968933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.949783 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:05.995265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:05.979246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27cj\" (UniqueName: \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\") pod \"maas-api-revoke-sub-6ec66e5e-rxltb\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:06.227246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:06.227219 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:06.693948 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:06.693911 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b60aede_e716_4ae0_baac_759cc76a4e32.slice/crio-8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a WatchSource:0}: Error finding container 8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a: Status 404 returned error can't find the container with id 8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a Oct 29 22:14:06.696272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:06.695923 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb"] Oct 29 22:14:07.604566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:07.603342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" event={"ID":"9b60aede-e716-4ae0-baac-759cc76a4e32","Type":"ContainerStarted","Data":"4c853e1edd7edc2235f33dcc484748456510aaa8036f190933a2420f4de5510e"} Oct 29 22:14:07.604566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:07.603408 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" event={"ID":"9b60aede-e716-4ae0-baac-759cc76a4e32","Type":"ContainerStarted","Data":"8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a"} Oct 29 22:14:07.695103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:07.694432 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" podStartSLOduration=2.694410998 podStartE2EDuration="2.694410998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:07.681531767 +0000 UTC m=+1909.423462441" watchObservedRunningTime="2026-10-29 22:14:07.694410998 +0000 UTC m=+1909.436341674" Oct 29 22:14:08.613885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:08.613849 4543 generic.go:356] "Generic (PLEG): container finished" podID="9b60aede-e716-4ae0-baac-759cc76a4e32" containerID="4c853e1edd7edc2235f33dcc484748456510aaa8036f190933a2420f4de5510e" exitCode=0 Oct 29 22:14:08.614099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:08.613899 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" event={"ID":"9b60aede-e716-4ae0-baac-759cc76a4e32","Type":"ContainerDied","Data":"4c853e1edd7edc2235f33dcc484748456510aaa8036f190933a2420f4de5510e"} Oct 29 22:14:10.127191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.127159 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:10.222143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.222111 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\" (UniqueName: \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\") pod \"9b60aede-e716-4ae0-baac-759cc76a4e32\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " Oct 29 22:14:10.222329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.222194 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\") pod \"9b60aede-e716-4ae0-baac-759cc76a4e32\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " Oct 29 22:14:10.222809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.222504 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\") pod \"9b60aede-e716-4ae0-baac-759cc76a4e32\" (UID: \"9b60aede-e716-4ae0-baac-759cc76a4e32\") " Oct 29 22:14:10.222809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.222592 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca" pod "9b60aede-e716-4ae0-baac-759cc76a4e32" (UID: "9b60aede-e716-4ae0-baac-759cc76a4e32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:14:10.223270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.223245 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:10.227359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.227326 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca" pod "9b60aede-e716-4ae0-baac-759cc76a4e32" (UID: "9b60aede-e716-4ae0-baac-759cc76a4e32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:14:10.227555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.227532 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj" pod "9b60aede-e716-4ae0-baac-759cc76a4e32" (UID: "9b60aede-e716-4ae0-baac-759cc76a4e32"). InnerVolumeSpecName "kube-api-access-x27cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:10.324299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.324273 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27cj\" (UniqueName: \"kubernetes.io/projected/9b60aede-e716-4ae0-baac-759cc76a4e32-kube-api-access-x27cj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:10.324299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.324298 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b60aede-e716-4ae0-baac-759cc76a4e32-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:14:10.660128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.660057 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" Oct 29 22:14:10.660128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.660102 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb" event={"ID":"9b60aede-e716-4ae0-baac-759cc76a4e32","Type":"ContainerDied","Data":"8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a"} Oct 29 22:14:10.660340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:10.660163 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b46a7126880cb3cb2cc599824c88549d5a4f01b199e830f7725c6ad6aeb149a" Oct 29 22:14:12.031989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.031949 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:14:12.033238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.033219 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9b60aede-e716-4ae0-baac-759cc76a4e32" containerName="revoke-keys" Oct 29 22:14:12.033337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.033240 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b60aede-e716-4ae0-baac-759cc76a4e32" containerName="revoke-keys" Oct 29 22:14:12.033452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.033438 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b60aede-e716-4ae0-baac-759cc76a4e32" containerName="revoke-keys" Oct 29 22:14:12.038252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.038233 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.055552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.055526 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:14:12.060312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.060265 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4afaba\"" Oct 29 22:14:12.060312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.060308 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4afaba-dockercfg-bf2z8\"" Oct 29 22:14:12.144100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.139405 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.214115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.203805 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9rj\" (UniqueName: \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.315177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.307520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9rj\" (UniqueName: \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.332704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.315418 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:14:12.332704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.330859 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.332704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.331339 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.337914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.337892 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.350659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.350623 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:14:12.360098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.360008 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9rj\" (UniqueName: \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\") pod \"payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.434562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.434526 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qvj\" (UniqueName: \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.434768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.434687 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.535934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.535889 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qvj\" (UniqueName: \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.536177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.536094 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.537187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.537160 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.562431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.562392 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qvj\" (UniqueName: \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\") pod \"payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.658178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.658092 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:12.667248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.667219 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:12.986219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:12.986174 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:14:12.987337 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:12.987303 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2be3143_8369_43b8_83e3_99e21b36903d.slice/crio-bebf37470a1164bd9bb1040ed89f16da24cfb93b6aa51b22b831ba806acb1999 WatchSource:0}: Error finding container bebf37470a1164bd9bb1040ed89f16da24cfb93b6aa51b22b831ba806acb1999: Status 404 returned error can't find the container with id bebf37470a1164bd9bb1040ed89f16da24cfb93b6aa51b22b831ba806acb1999 Oct 29 22:14:13.016158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.016133 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:14:13.886703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.886644 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" event={"ID":"78493013-3db7-46a0-a3c0-bce2b9050541","Type":"ContainerStarted","Data":"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273"} Oct 29 22:14:13.886703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.886708 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" event={"ID":"78493013-3db7-46a0-a3c0-bce2b9050541","Type":"ContainerStarted","Data":"111fc8e43e4cf9c6ddd902c807ab62ce9d7dc233507b79bd943ed1a1b0786637"} Oct 29 22:14:13.887900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.887878 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:13.898428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.898385 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" event={"ID":"a2be3143-8369-43b8-83e3-99e21b36903d","Type":"ContainerStarted","Data":"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6"} Oct 29 22:14:13.898572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.898441 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" event={"ID":"a2be3143-8369-43b8-83e3-99e21b36903d","Type":"ContainerStarted","Data":"bebf37470a1164bd9bb1040ed89f16da24cfb93b6aa51b22b831ba806acb1999"} Oct 29 22:14:13.899346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.899280 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:13.938598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.938516 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" podStartSLOduration=1.9384986880000001 podStartE2EDuration="1.938498688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.93101261 +0000 UTC m=+1915.672943285" watchObservedRunningTime="2026-10-29 22:14:13.938498688 +0000 UTC m=+1915.680429362" Oct 29 22:14:13.977917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:13.977862 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" podStartSLOduration=2.977844134 podStartE2EDuration="2.977844134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.974847771 +0000 UTC m=+1915.716778444" watchObservedRunningTime="2026-10-29 22:14:13.977844134 +0000 UTC m=+1915.719774808" Oct 29 22:14:15.920103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.920054 4543 generic.go:356] "Generic (PLEG): container finished" podID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerID="0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59" exitCode=0 Oct 29 22:14:15.920594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.920108 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerDied","Data":"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59"} Oct 29 22:14:15.920594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.920552 4543 scope.go:122] "RemoveContainer" containerID="0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59" Oct 29 22:14:15.922254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922235 4543 generic.go:356] "Generic (PLEG): container finished" podID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerID="7c9e54eaf236ac6a6e3f297d83f26dde56fe9be33dc15b505a3642ea27ecaddd" exitCode=0 Oct 29 22:14:15.922365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922308 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerDied","Data":"7c9e54eaf236ac6a6e3f297d83f26dde56fe9be33dc15b505a3642ea27ecaddd"} Oct 29 22:14:15.922414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922314 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:15.922478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922457 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:15.922529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922513 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:15.922924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.922909 4543 scope.go:122] "RemoveContainer" containerID="7c9e54eaf236ac6a6e3f297d83f26dde56fe9be33dc15b505a3642ea27ecaddd" Oct 29 22:14:15.924436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.924413 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:15.924519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.924471 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:15.924519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:15.924503 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:16.556295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:16.556264 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:14:16.557291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:16.557271 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:14:16.893576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:16.893476 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:14:16.975477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:16.975442 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerStarted","Data":"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81"} Oct 29 22:14:16.979483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:16.979441 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerStarted","Data":"1e9ca86ba0c58ef10f2012b0c9c9ff1764809dcc4986dd105667754ca61f3446"} Oct 29 22:14:17.294131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.292909 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:14:17.294365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.294343 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:17.294421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.294396 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:17.750101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.746461 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:14:17.750101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.748279 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:17.750101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:17.748338 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:18.292554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.292518 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:18.292989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.292571 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:18.757203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.757164 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:18.761199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.761140 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:18.913229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.913198 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:14:18.916976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.916950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:18.925386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.925167 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:18.925386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.925361 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gl9\" (UniqueName: \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:18.974103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.963791 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8d569e\"" Oct 29 22:14:18.974103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:18.963866 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8d569e-dockercfg-hw2nn\"" Oct 29 22:14:19.042147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.034215 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gl9\" (UniqueName: \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:19.042147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.034406 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:19.042147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.035551 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:19.156823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.052276 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:14:19.156823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.146279 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gl9\" (UniqueName: \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\") pod \"payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:19.302750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.302662 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:19.311415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.311381 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:19.325593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.325562 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:19.767102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.766101 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:19.767102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.766163 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:19.783122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:19.782914 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:14:19.784643 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:19.784593 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2abb55_3000_4980_ad69_1bd1ada616ee.slice/crio-ab3984ac4d2a1e107b224582d9dd38cec38ccf884be704f9e7e23e044213e326 WatchSource:0}: Error finding container ab3984ac4d2a1e107b224582d9dd38cec38ccf884be704f9e7e23e044213e326: Status 404 returned error can't find the container with id ab3984ac4d2a1e107b224582d9dd38cec38ccf884be704f9e7e23e044213e326 Oct 29 22:14:20.222533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.222493 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" event={"ID":"3b2abb55-3000-4980-ad69-1bd1ada616ee","Type":"ContainerStarted","Data":"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0"} Oct 29 22:14:20.222709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.222586 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" event={"ID":"3b2abb55-3000-4980-ad69-1bd1ada616ee","Type":"ContainerStarted","Data":"ab3984ac4d2a1e107b224582d9dd38cec38ccf884be704f9e7e23e044213e326"} Oct 29 22:14:20.222709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.222692 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:20.284807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.284644 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" podStartSLOduration=2.284629402 podStartE2EDuration="2.284629402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:20.268427541 +0000 UTC m=+1922.010358215" watchObservedRunningTime="2026-10-29 22:14:20.284629402 +0000 UTC m=+1922.026560074" Oct 29 22:14:20.297408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.297374 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:20.297571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.297425 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:20.752102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.748489 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:20.752102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:20.748554 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:21.292881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.292843 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:21.293137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.292911 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:21.752104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.749461 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:21.752104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.749521 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:21.905145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.903495 4543 scope.go:122] "RemoveContainer" containerID="6f3fd21f1be66caa3f3be45f89a1b781fde71dd20501de456ace17a4c5139969" Oct 29 22:14:21.921039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.921013 4543 scope.go:122] "RemoveContainer" containerID="c8f03a4a9736784e7ee8d759f2e463afea6b7f5c006489d779ffeb3760dd088c" Oct 29 22:14:21.974125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:21.972375 4543 scope.go:122] "RemoveContainer" containerID="275718283c32612f44a226f2e4e10a92c05530d904379595ed70cf62f94de545" Oct 29 22:14:22.015567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.015542 4543 scope.go:122] "RemoveContainer" containerID="5f9be9b7cd3a5c8232b095c4f125a2944e23d58dde7cc4fae17f209830d0f9e4" Oct 29 22:14:22.026778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.026709 4543 scope.go:122] "RemoveContainer" containerID="e6b79be0caae321bede3b44234be34f0b9a45fcea5b6b01d7e3d620ac8b32b9a" Oct 29 22:14:22.293330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.293249 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 22:14:22.293330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.293309 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 22:14:22.316308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.316283 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:14:22.746411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.746378 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 29 22:14:22.746590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:22.746434 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 29 22:14:23.297160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:23.297134 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:14:23.297613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:23.297537 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:14:23.298249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:23.298231 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:14:23.751256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:23.751232 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:14:23.752323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:23.752300 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:14:25.925062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:25.925037 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:14:25.925474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:25.925139 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:14:26.920745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.920713 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:14:26.926176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.926155 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956657 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q9j\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956833 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.956971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.956959 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.957011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.957004 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.957068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.957028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:26.967970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:26.967947 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:14:27.058605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058567 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.058788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058666 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q9j\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.058788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058705 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.058788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058762 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.058952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058800 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.058952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.058944 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.059089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.059002 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.059441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.059368 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.059441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.059397 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.059730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.059707 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.059936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.059914 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.060070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.060034 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.060355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.060336 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.060429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.060402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.061930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.061908 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.062187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.062166 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.084788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.084764 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.084889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.084784 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q9j\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.194851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.194769 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:14:27.199188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.199165 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.200009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.199986 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:14:27.203412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.203393 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.216508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.216487 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0a3d53\"" Oct 29 22:14:27.216866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.216844 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0a3d53-dockercfg-tr2wg\"" Oct 29 22:14:27.219379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.219354 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:14:27.224847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.224825 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:14:27.239619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.239600 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:27.268198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.268156 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd5z\" (UniqueName: \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.268527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.268506 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.268929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.268908 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.269156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.269137 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsbv\" (UniqueName: \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.426100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.416181 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd5z\" (UniqueName: \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.426100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.416473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.426100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.416699 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.426100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.416755 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsbv\" (UniqueName: \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.449147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.427398 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.449147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.428173 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.487106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.480698 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd5z\" (UniqueName: \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\") pod \"payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.496097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.487595 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsbv\" (UniqueName: \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\") pod \"payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.519696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.518054 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:27.528027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.527997 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:27.811166 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:27.811133 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2179d4a_0713_4c8c_b1cd_1068b8d9ba3f.slice/crio-e78c4e570603dabe2f907744cd2cb056acc36d41381ca0eb8fda97cb12aca125 WatchSource:0}: Error finding container e78c4e570603dabe2f907744cd2cb056acc36d41381ca0eb8fda97cb12aca125: Status 404 returned error can't find the container with id e78c4e570603dabe2f907744cd2cb056acc36d41381ca0eb8fda97cb12aca125 Oct 29 22:14:27.811856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.811832 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:14:27.815745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.815711 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:27.815812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.815782 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:27.815917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.815818 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:27.913129 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:27.912764 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca88882_9d81_4711_8c2f_320d1ada85f4.slice/crio-c4113b436cee4a8381d6a8cd404aa13b91c20f198941a36d705ec0203196cd7b WatchSource:0}: Error finding container c4113b436cee4a8381d6a8cd404aa13b91c20f198941a36d705ec0203196cd7b: Status 404 returned error can't find the container with id c4113b436cee4a8381d6a8cd404aa13b91c20f198941a36d705ec0203196cd7b Oct 29 22:14:27.918425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.918404 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:14:27.925269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:27.924947 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:14:27.925553 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:27.925527 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030e8810_2efb_49bf_a641_2bd99adc75e4.slice/crio-08962bb8998403fb03e9d64e06f90210464dae0bd9ddbc6649646da55304e7c1 WatchSource:0}: Error finding container 08962bb8998403fb03e9d64e06f90210464dae0bd9ddbc6649646da55304e7c1: Status 404 returned error can't find the container with id 08962bb8998403fb03e9d64e06f90210464dae0bd9ddbc6649646da55304e7c1 Oct 29 22:14:27.975884 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:14:27.975852 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:14:28.500916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.500881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" event={"ID":"fca88882-9d81-4711-8c2f-320d1ada85f4","Type":"ContainerStarted","Data":"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a"} Oct 29 22:14:28.501168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.500943 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" event={"ID":"fca88882-9d81-4711-8c2f-320d1ada85f4","Type":"ContainerStarted","Data":"c4113b436cee4a8381d6a8cd404aa13b91c20f198941a36d705ec0203196cd7b"} Oct 29 22:14:28.501168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.500972 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:28.502795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.502763 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" event={"ID":"030e8810-2efb-49bf-a641-2bd99adc75e4","Type":"ContainerStarted","Data":"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b"} Oct 29 22:14:28.502942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.502818 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" event={"ID":"030e8810-2efb-49bf-a641-2bd99adc75e4","Type":"ContainerStarted","Data":"08962bb8998403fb03e9d64e06f90210464dae0bd9ddbc6649646da55304e7c1"} Oct 29 22:14:28.502942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.502846 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:28.504418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.504395 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" event={"ID":"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f","Type":"ContainerStarted","Data":"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569"} Oct 29 22:14:28.504535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.504441 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" event={"ID":"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f","Type":"ContainerStarted","Data":"e78c4e570603dabe2f907744cd2cb056acc36d41381ca0eb8fda97cb12aca125"} Oct 29 22:14:28.555477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.555298 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" podStartSLOduration=1.555278414 podStartE2EDuration="1.555278414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.547532423 +0000 UTC m=+1930.289463098" watchObservedRunningTime="2026-10-29 22:14:28.555278414 +0000 UTC m=+1930.297209087" Oct 29 22:14:28.601657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.601611 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podStartSLOduration=2.601597214 podStartE2EDuration="2.601597214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.595852733 +0000 UTC m=+1930.337783420" watchObservedRunningTime="2026-10-29 22:14:28.601597214 +0000 UTC m=+1930.343527886" Oct 29 22:14:28.657148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:28.656723 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" podStartSLOduration=1.656701204 podStartE2EDuration="1.656701204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.644949928 +0000 UTC m=+1930.386880603" watchObservedRunningTime="2026-10-29 22:14:28.656701204 +0000 UTC m=+1930.398631878" Oct 29 22:14:29.253164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:29.250429 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:29.255046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:29.255014 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:29.255192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:29.255089 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:30.242447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:30.242239 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:30.242447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:30.242302 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:31.058918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:31.058882 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh"] Oct 29 22:14:31.064134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:31.064105 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e897b851-vgxzh"] Oct 29 22:14:31.244098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:31.243516 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:31.244098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:31.243581 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:31.293340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:31.293318 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:14:32.059975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.059937 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn"] Oct 29 22:14:32.064667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.064643 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57326934-pnpbn"] Oct 29 22:14:32.242111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.241410 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:32.242111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.241485 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:32.387527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.387446 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758943d0-7f41-43d5-ab7f-e45b39087124" path="/var/lib/kubelet/pods/758943d0-7f41-43d5-ab7f-e45b39087124/volumes" Oct 29 22:14:32.387889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:32.387874 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d8003a-89e1-45dc-891b-f2dff5b62035" path="/var/lib/kubelet/pods/c6d8003a-89e1-45dc-891b-f2dff5b62035/volumes" Oct 29 22:14:33.247479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:33.247445 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:33.248087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:33.248038 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:34.240333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:34.240298 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 29 22:14:34.240518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:34.240349 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 29 22:14:35.244974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:35.244942 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:35.245503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:35.245480 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:35.246168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:35.246151 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:14:39.516308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:39.516276 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:14:39.516907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:39.516341 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:14:41.934154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:41.934121 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:14:41.938282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:41.938261 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:41.984119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:41.980721 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.076209 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp"] Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.077376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.077426 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.077511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.077538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59qrc\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.078100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.077699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.079615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.079586 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.080023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.079985 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.087302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.087272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.093299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.093271 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.101863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.101835 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58362f62-qcmjp"] Oct 29 22:14:42.195357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195272 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195312 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195478 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195600 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195632 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195711 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195763 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59qrc\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195924 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.195948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.195926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.196185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.196133 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.196185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.196171 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.196416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.196264 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.196485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.196463 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.198192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.198172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.198415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.198397 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.221240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.221219 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.221578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.221560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59qrc\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.254375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.254352 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:42.393989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.393951 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af08d05c-7f5b-479f-b795-de095602ac80" path="/var/lib/kubelet/pods/af08d05c-7f5b-479f-b795-de095602ac80/volumes" Oct 29 22:14:42.581462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.581437 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:14:42.581784 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:14:42.581760 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc224878_39b1_4043_9d1c_a2f6b607803a.slice/crio-62f062add873d6d2513aee4fd68b697cd8021c7a41f6b5a3bbdc8a69412d7137 WatchSource:0}: Error finding container 62f062add873d6d2513aee4fd68b697cd8021c7a41f6b5a3bbdc8a69412d7137: Status 404 returned error can't find the container with id 62f062add873d6d2513aee4fd68b697cd8021c7a41f6b5a3bbdc8a69412d7137 Oct 29 22:14:42.584698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.584655 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:42.584824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.584743 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:42.584824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.584781 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:14:42.679505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.679475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" event={"ID":"bc224878-39b1-4043-9d1c-a2f6b607803a","Type":"ContainerStarted","Data":"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024"} Oct 29 22:14:42.679619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.679527 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" event={"ID":"bc224878-39b1-4043-9d1c-a2f6b607803a","Type":"ContainerStarted","Data":"62f062add873d6d2513aee4fd68b697cd8021c7a41f6b5a3bbdc8a69412d7137"} Oct 29 22:14:42.734046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:42.733982 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podStartSLOduration=1.733962972 podStartE2EDuration="1.733962972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:42.726876022 +0000 UTC m=+1944.468806695" watchObservedRunningTime="2026-10-29 22:14:42.733962972 +0000 UTC m=+1944.475893647" Oct 29 22:14:43.254821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:43.254788 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:43.256312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:43.256284 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:43.256448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:43.256343 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:44.059333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:44.059298 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52"] Oct 29 22:14:44.063981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:44.063955 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79e38bbc-rmp52"] Oct 29 22:14:44.255659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:44.255609 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:44.256244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:44.255675 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:44.403044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:44.402968 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6109265-a586-4622-90eb-2a75272e6a37" path="/var/lib/kubelet/pods/c6109265-a586-4622-90eb-2a75272e6a37/volumes" Oct 29 22:14:45.255581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:45.255543 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:45.255781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:45.255613 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:46.280842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:46.280794 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:46.281581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:46.281552 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:47.258102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:47.257364 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:47.258102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:47.257420 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:48.254939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:48.254902 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:48.255384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:48.254956 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:49.254947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:49.254911 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 29 22:14:49.255345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:49.254963 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 29 22:14:51.296228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:51.296191 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:14:51.296631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:51.296256 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:14:51.775535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:51.775502 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:51.776039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:51.776011 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:51.776861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:51.776841 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:14:55.039008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:55.038974 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5"] Oct 29 22:14:55.050713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:55.050687 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecdb9c5c-6kbn5"] Oct 29 22:14:56.387754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:14:56.387722 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28110085-6c20-44d7-b72a-3d3b28a76440" path="/var/lib/kubelet/pods/28110085-6c20-44d7-b72a-3d3b28a76440/volumes" Oct 29 22:14:57.961922 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:14:57.961882 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:15:00.146749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.146710 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:00.151635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.151614 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.151879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.151861 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:00.157900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.157880 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:00.158029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.157979 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.162181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.162162 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.162343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.162284 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:00.166352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.166190 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.169200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.169179 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:00.175464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.175446 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:00.175620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.175602 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.180791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.180771 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.180954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.180935 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:15:00.181065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.181005 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:15:00.182964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.182360 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:00.184069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.184043 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:15:00.189562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.189536 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:00.217380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217269 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6v9\" (UniqueName: \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.217567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217387 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.217567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217438 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdf6\" (UniqueName: \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.217567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217531 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.217781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217595 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.217781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217680 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmgq\" (UniqueName: \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.217781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217725 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.217781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217762 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.217942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217808 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.217942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj98c\" (UniqueName: \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.217942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.217888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmwn\" (UniqueName: \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.218096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.218030 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72f96\" (UniqueName: \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.226704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.226123 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:00.261298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.235884 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.261298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.241527 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:00.261298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.250944 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:00.277407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.277373 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:00.295249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.294794 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:00.296594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.296575 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.304710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.304684 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:00.304855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.304839 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.311339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.311317 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:00.316670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.316645 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.320088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.320202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320114 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72f96\" (UniqueName: \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.320202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320174 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6v9\" (UniqueName: \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.320331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gpz\" (UniqueName: \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.320331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320267 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.320331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320304 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdf6\" (UniqueName: \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.320496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320395 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.320496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320431 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.320496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320481 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.320645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.320567 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvdw\" (UniqueName: \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.321128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.321106 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.322169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322143 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmgq\" (UniqueName: \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.322253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322211 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.322253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322239 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.322333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.322438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322412 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.323014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.322992 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.323091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.323013 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.323947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.323919 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.326877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.326854 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj98c\" (UniqueName: \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.326988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.326905 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmwn\" (UniqueName: \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.326988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.326953 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjkp\" (UniqueName: \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.326988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.326984 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.328048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.328014 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.328192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.328143 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:00.336946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.336363 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:00.343385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.343318 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:00.349976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.349955 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:00.355117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.353721 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6v9\" (UniqueName: \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.356653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.356629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdf6\" (UniqueName: \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.356870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.356803 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72f96\" (UniqueName: \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.357735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.357697 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmwn\" (UniqueName: \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.357863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.357844 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:00.357962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.357948 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.358433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.358395 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmgq\" (UniqueName: \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.360028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.360003 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj98c\" (UniqueName: \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.363326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.363309 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:00.382611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.375472 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:00.428711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.428679 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.428882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.428791 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gpz\" (UniqueName: \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.428882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.428851 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.429013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.428973 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.429013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429001 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvmg\" (UniqueName: \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.429235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429148 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvdw\" (UniqueName: \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.429305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429241 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khcsk\" (UniqueName: \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.429369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429328 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.429424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429376 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjkp\" (UniqueName: \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.429424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429408 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.429547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429504 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.429852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.429828 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.430573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.430550 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.458441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.458418 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvdw\" (UniqueName: \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.458664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.458641 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gpz\" (UniqueName: \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.458723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.458701 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjkp\" (UniqueName: \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.477159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.477141 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:00.530772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.530738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.530944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.530856 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvmg\" (UniqueName: \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.531023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.530978 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khcsk\" (UniqueName: \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.531103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.531038 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.531660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.531625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.538121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.533070 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.541971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.541947 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:00.559661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.559631 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvmg\" (UniqueName: \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.561979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.561600 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khcsk\" (UniqueName: \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.577915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.577513 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:00.625724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.625441 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:00.656856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.654555 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:00.656856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.654703 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:00.656856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.654773 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:00.656856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.654833 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:00.656856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.654891 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:00.719429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.719371 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:00.726780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.726695 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:00.794946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.794902 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:00.845672 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:00.845643 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec53711c_2418_4464_bd4a_2ca258fe10e3.slice/crio-0bccd17dd9f979f1760742ed8724be14a57fa756d5a22fcc46ed1c29f1638365 WatchSource:0}: Error finding container 0bccd17dd9f979f1760742ed8724be14a57fa756d5a22fcc46ed1c29f1638365: Status 404 returned error can't find the container with id 0bccd17dd9f979f1760742ed8724be14a57fa756d5a22fcc46ed1c29f1638365 Oct 29 22:15:00.983965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.983939 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:00.987632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:00.987563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerStarted","Data":"0bccd17dd9f979f1760742ed8724be14a57fa756d5a22fcc46ed1c29f1638365"} Oct 29 22:15:01.020087 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.019997 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ce399b_2b7f_4c84_8362_89a4efcfca34.slice/crio-e084384671e606f6e3246b982c3c62f14467b7ff11508bae0d29d62be22184c9 WatchSource:0}: Error finding container e084384671e606f6e3246b982c3c62f14467b7ff11508bae0d29d62be22184c9: Status 404 returned error can't find the container with id e084384671e606f6e3246b982c3c62f14467b7ff11508bae0d29d62be22184c9 Oct 29 22:15:01.112928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.111554 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:01.169624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.169587 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:01.234367 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.234334 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71d77a0_9414_40e6_8237_5cb0d7697a41.slice/crio-1bcd3c9bc145a09e00dbf48aa9fd8945914df936f63b1a8a07df47e6a93d069c WatchSource:0}: Error finding container 1bcd3c9bc145a09e00dbf48aa9fd8945914df936f63b1a8a07df47e6a93d069c: Status 404 returned error can't find the container with id 1bcd3c9bc145a09e00dbf48aa9fd8945914df936f63b1a8a07df47e6a93d069c Oct 29 22:15:01.391803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.391779 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:01.403486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.403446 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:01.434131 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.434101 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b12988_8477_44bb_a903_d780202fb9e0.slice/crio-1edf6a5e8356cf1d3e7782f73906e9ec7eb70f0f4183c2b476969d438db9d34b WatchSource:0}: Error finding container 1edf6a5e8356cf1d3e7782f73906e9ec7eb70f0f4183c2b476969d438db9d34b: Status 404 returned error can't find the container with id 1edf6a5e8356cf1d3e7782f73906e9ec7eb70f0f4183c2b476969d438db9d34b Oct 29 22:15:01.434833 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.434795 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8185d5_4592_4761_b078_7cd63d635020.slice/crio-7ad47615fbfb18d481545f6a4a3e2a46724ae1f84807936ffb94a38bf923e353 WatchSource:0}: Error finding container 7ad47615fbfb18d481545f6a4a3e2a46724ae1f84807936ffb94a38bf923e353: Status 404 returned error can't find the container with id 7ad47615fbfb18d481545f6a4a3e2a46724ae1f84807936ffb94a38bf923e353 Oct 29 22:15:01.468518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.468410 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:01.477707 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.477642 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f181c_5a88_44f7_886f_c255f46319c4.slice/crio-efa13f6719d452e2f1cae2131f65f17a551a8cd036fa85dcbcc52e81e3ee2f15 WatchSource:0}: Error finding container efa13f6719d452e2f1cae2131f65f17a551a8cd036fa85dcbcc52e81e3ee2f15: Status 404 returned error can't find the container with id efa13f6719d452e2f1cae2131f65f17a551a8cd036fa85dcbcc52e81e3ee2f15 Oct 29 22:15:01.477957 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.477925 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04542cbe_3637_4338_a594_49c7406422bf.slice/crio-3eedf44b141a624142621f1dbcc83646447562fbff8e4cd5e51c3ee8065d10c9 WatchSource:0}: Error finding container 3eedf44b141a624142621f1dbcc83646447562fbff8e4cd5e51c3ee8065d10c9: Status 404 returned error can't find the container with id 3eedf44b141a624142621f1dbcc83646447562fbff8e4cd5e51c3ee8065d10c9 Oct 29 22:15:01.478470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.478408 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:01.486167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.486144 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:01.527537 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.527511 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d4c4a3_deda_4ad9_a481_d7bf244440cd.slice/crio-99cb615be1d21af2ac02939bccd8062b8a2ade7db5afac9f77dc42a15f11203e WatchSource:0}: Error finding container 99cb615be1d21af2ac02939bccd8062b8a2ade7db5afac9f77dc42a15f11203e: Status 404 returned error can't find the container with id 99cb615be1d21af2ac02939bccd8062b8a2ade7db5afac9f77dc42a15f11203e Oct 29 22:15:01.527950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.527928 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:01.557330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.557308 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:01.561774 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:01.561747 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2f1188_e008_45a9_b06c_6bcb05625eec.slice/crio-61692630de3a88b55fe2dc84fdada9bd678c3acfbd2cc3dae38c236132f7fe2b WatchSource:0}: Error finding container 61692630de3a88b55fe2dc84fdada9bd678c3acfbd2cc3dae38c236132f7fe2b: Status 404 returned error can't find the container with id 61692630de3a88b55fe2dc84fdada9bd678c3acfbd2cc3dae38c236132f7fe2b Oct 29 22:15:01.998718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:01.998662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerStarted","Data":"7ad47615fbfb18d481545f6a4a3e2a46724ae1f84807936ffb94a38bf923e353"} Oct 29 22:15:02.000142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.000108 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerStarted","Data":"99cb615be1d21af2ac02939bccd8062b8a2ade7db5afac9f77dc42a15f11203e"} Oct 29 22:15:02.001758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.001733 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerStarted","Data":"25b2c14b521423ab98ea280396ce969e12cbbaed84dfe6e7c473981fa36f6641"} Oct 29 22:15:02.001863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.001778 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerStarted","Data":"e084384671e606f6e3246b982c3c62f14467b7ff11508bae0d29d62be22184c9"} Oct 29 22:15:02.003202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.003179 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerStarted","Data":"3eedf44b141a624142621f1dbcc83646447562fbff8e4cd5e51c3ee8065d10c9"} Oct 29 22:15:02.004529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.004505 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerStarted","Data":"c22e83a0768e4f6343eae565e2c67c4903bcaa0a7f2c8a1ce08aa3b526978b43"} Oct 29 22:15:02.005835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.005814 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerStarted","Data":"61692630de3a88b55fe2dc84fdada9bd678c3acfbd2cc3dae38c236132f7fe2b"} Oct 29 22:15:02.007381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.007356 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerStarted","Data":"4b652b576bd625956ca0b15a48f04a8ce29ca9474e341e5d042f8cd547c8ffda"} Oct 29 22:15:02.007486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.007402 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerStarted","Data":"1bcd3c9bc145a09e00dbf48aa9fd8945914df936f63b1a8a07df47e6a93d069c"} Oct 29 22:15:02.008643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.008621 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerStarted","Data":"efa13f6719d452e2f1cae2131f65f17a551a8cd036fa85dcbcc52e81e3ee2f15"} Oct 29 22:15:02.009838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.009815 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerStarted","Data":"1edf6a5e8356cf1d3e7782f73906e9ec7eb70f0f4183c2b476969d438db9d34b"} Oct 29 22:15:02.011336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.011317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerStarted","Data":"5bd359f5bc7e847f83a3d17dfc9bb6e82713567ad538790d65728f0718688e22"} Oct 29 22:15:02.012956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.012935 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerStarted","Data":"4e6655356456672bcde531ed1489fb5da163e0387d0b7445b3ec1cb117fc769c"} Oct 29 22:15:02.013031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.012978 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerStarted","Data":"1ccdce50e9cbbcfbaa768230d7c27c2fa5d3760e5a035102310f7c4580a32d81"} Oct 29 22:15:02.046686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.046647 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" podStartSLOduration=2.046635428 podStartE2EDuration="2.046635428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.045326482 +0000 UTC m=+1963.787257156" watchObservedRunningTime="2026-10-29 22:15:02.046635428 +0000 UTC m=+1963.788566101" Oct 29 22:15:02.093889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.093826 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" podStartSLOduration=2.093808565 podStartE2EDuration="2.093808565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.090957116 +0000 UTC m=+1963.832887789" watchObservedRunningTime="2026-10-29 22:15:02.093808565 +0000 UTC m=+1963.835739239" Oct 29 22:15:02.142140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.142044 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" podStartSLOduration=2.142030074 podStartE2EDuration="2.142030074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.133565222 +0000 UTC m=+1963.875495896" watchObservedRunningTime="2026-10-29 22:15:02.142030074 +0000 UTC m=+1963.883960747" Oct 29 22:15:02.179157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:02.179111 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" podStartSLOduration=2.179098166 podStartE2EDuration="2.179098166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.178153261 +0000 UTC m=+1963.920083933" watchObservedRunningTime="2026-10-29 22:15:02.179098166 +0000 UTC m=+1963.921028839" Oct 29 22:15:03.024904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.024875 4543 generic.go:356] "Generic (PLEG): container finished" podID="04542cbe-3637-4338-a594-49c7406422bf" containerID="8d72039b7b21d1f64b140c28415180d95b577b02cbb1f43bbcec1ff4639b732f" exitCode=60 Oct 29 22:15:03.025038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.024959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerDied","Data":"8d72039b7b21d1f64b140c28415180d95b577b02cbb1f43bbcec1ff4639b732f"} Oct 29 22:15:03.025275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.025256 4543 scope.go:122] "RemoveContainer" containerID="8d72039b7b21d1f64b140c28415180d95b577b02cbb1f43bbcec1ff4639b732f" Oct 29 22:15:03.027065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.027047 4543 generic.go:356] "Generic (PLEG): container finished" podID="e2984b7c-87fe-42ce-a046-81066764caad" containerID="acf1da4ddb3cf10651623101f3ffa56e23816fadb15121dd04e5f8c6b9450aed" exitCode=60 Oct 29 22:15:03.027214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.027142 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerDied","Data":"acf1da4ddb3cf10651623101f3ffa56e23816fadb15121dd04e5f8c6b9450aed"} Oct 29 22:15:03.027432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.027409 4543 scope.go:122] "RemoveContainer" containerID="acf1da4ddb3cf10651623101f3ffa56e23816fadb15121dd04e5f8c6b9450aed" Oct 29 22:15:03.029164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.029142 4543 generic.go:356] "Generic (PLEG): container finished" podID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerID="243c5c339923b24795ea678209a67cf9f006253a61eea91e691913833edcdc20" exitCode=60 Oct 29 22:15:03.029234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.029186 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerDied","Data":"243c5c339923b24795ea678209a67cf9f006253a61eea91e691913833edcdc20"} Oct 29 22:15:03.029449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.029434 4543 scope.go:122] "RemoveContainer" containerID="243c5c339923b24795ea678209a67cf9f006253a61eea91e691913833edcdc20" Oct 29 22:15:03.031158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.031132 4543 generic.go:356] "Generic (PLEG): container finished" podID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerID="4b652b576bd625956ca0b15a48f04a8ce29ca9474e341e5d042f8cd547c8ffda" exitCode=60 Oct 29 22:15:03.031238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.031216 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerDied","Data":"4b652b576bd625956ca0b15a48f04a8ce29ca9474e341e5d042f8cd547c8ffda"} Oct 29 22:15:03.031427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.031411 4543 scope.go:122] "RemoveContainer" containerID="4b652b576bd625956ca0b15a48f04a8ce29ca9474e341e5d042f8cd547c8ffda" Oct 29 22:15:03.033013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.032991 4543 generic.go:356] "Generic (PLEG): container finished" podID="647f181c-5a88-44f7-886f-c255f46319c4" containerID="814b716b9721027cf479959dd7be1a1e5e2c63aa195d9f96a4fa8a71972951a4" exitCode=60 Oct 29 22:15:03.033138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.033052 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerDied","Data":"814b716b9721027cf479959dd7be1a1e5e2c63aa195d9f96a4fa8a71972951a4"} Oct 29 22:15:03.033360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.033341 4543 scope.go:122] "RemoveContainer" containerID="814b716b9721027cf479959dd7be1a1e5e2c63aa195d9f96a4fa8a71972951a4" Oct 29 22:15:03.034838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.034818 4543 generic.go:356] "Generic (PLEG): container finished" podID="64b12988-8477-44bb-a903-d780202fb9e0" containerID="6e26cec204df6e3d37b58ca48b6d38d2941bfa0e36bca6892d7195e6fc1793d9" exitCode=60 Oct 29 22:15:03.034932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.034845 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerDied","Data":"6e26cec204df6e3d37b58ca48b6d38d2941bfa0e36bca6892d7195e6fc1793d9"} Oct 29 22:15:03.035135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.035119 4543 scope.go:122] "RemoveContainer" containerID="6e26cec204df6e3d37b58ca48b6d38d2941bfa0e36bca6892d7195e6fc1793d9" Oct 29 22:15:03.036714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.036685 4543 generic.go:356] "Generic (PLEG): container finished" podID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerID="5bd359f5bc7e847f83a3d17dfc9bb6e82713567ad538790d65728f0718688e22" exitCode=60 Oct 29 22:15:03.036797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.036744 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerDied","Data":"5bd359f5bc7e847f83a3d17dfc9bb6e82713567ad538790d65728f0718688e22"} Oct 29 22:15:03.037015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.036999 4543 scope.go:122] "RemoveContainer" containerID="5bd359f5bc7e847f83a3d17dfc9bb6e82713567ad538790d65728f0718688e22" Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.039259 4543 generic.go:356] "Generic (PLEG): container finished" podID="71348e81-9a5d-4738-ad0c-561ea7442536" containerID="4e6655356456672bcde531ed1489fb5da163e0387d0b7445b3ec1cb117fc769c" exitCode=60 Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.039481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerDied","Data":"4e6655356456672bcde531ed1489fb5da163e0387d0b7445b3ec1cb117fc769c"} Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.042214 4543 generic.go:356] "Generic (PLEG): container finished" podID="da8185d5-4592-4761-b078-7cd63d635020" containerID="d046f30733f1f4b7f66571af422d532706d0591ce396952dbc5a223276c1cfd8" exitCode=60 Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.042348 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerDied","Data":"d046f30733f1f4b7f66571af422d532706d0591ce396952dbc5a223276c1cfd8"} Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.042717 4543 scope.go:122] "RemoveContainer" containerID="d046f30733f1f4b7f66571af422d532706d0591ce396952dbc5a223276c1cfd8" Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.044330 4543 scope.go:122] "RemoveContainer" containerID="4e6655356456672bcde531ed1489fb5da163e0387d0b7445b3ec1cb117fc769c" Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.045657 4543 generic.go:356] "Generic (PLEG): container finished" podID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerID="7a4459aa59c38c80b9a4ec5efd5d870ec789444d84d85dd7f57c467029476298" exitCode=60 Oct 29 22:15:03.045892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.045774 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerDied","Data":"7a4459aa59c38c80b9a4ec5efd5d870ec789444d84d85dd7f57c467029476298"} Oct 29 22:15:03.046910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.046884 4543 scope.go:122] "RemoveContainer" containerID="7a4459aa59c38c80b9a4ec5efd5d870ec789444d84d85dd7f57c467029476298" Oct 29 22:15:03.048765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.048518 4543 generic.go:356] "Generic (PLEG): container finished" podID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerID="25b2c14b521423ab98ea280396ce969e12cbbaed84dfe6e7c473981fa36f6641" exitCode=60 Oct 29 22:15:03.048765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.048644 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerDied","Data":"25b2c14b521423ab98ea280396ce969e12cbbaed84dfe6e7c473981fa36f6641"} Oct 29 22:15:03.049346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.049324 4543 scope.go:122] "RemoveContainer" containerID="25b2c14b521423ab98ea280396ce969e12cbbaed84dfe6e7c473981fa36f6641" Oct 29 22:15:03.057973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.057954 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547"] Oct 29 22:15:03.067458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:03.067434 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7257ec0f-wr547"] Oct 29 22:15:04.059858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.059827 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerStarted","Data":"03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a"} Oct 29 22:15:04.062087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.062038 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerStarted","Data":"6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df"} Oct 29 22:15:04.064063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.064042 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerStarted","Data":"b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a"} Oct 29 22:15:04.066197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.066174 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerStarted","Data":"4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e"} Oct 29 22:15:04.068221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.068192 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerStarted","Data":"1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9"} Oct 29 22:15:04.070127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.070105 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerStarted","Data":"0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954"} Oct 29 22:15:04.071953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.071933 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerStarted","Data":"c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f"} Oct 29 22:15:04.073966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.073932 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerStarted","Data":"032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507"} Oct 29 22:15:04.076213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.076187 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerStarted","Data":"31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df"} Oct 29 22:15:04.078194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.078167 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerStarted","Data":"9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57"} Oct 29 22:15:04.079999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.079979 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerStarted","Data":"3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992"} Oct 29 22:15:04.192005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.191921 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" podStartSLOduration=4.19190789 podStartE2EDuration="4.19190789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.187069122 +0000 UTC m=+1965.928999794" watchObservedRunningTime="2026-10-29 22:15:04.19190789 +0000 UTC m=+1965.933838563" Oct 29 22:15:04.236205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.236153 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" podStartSLOduration=4.236137501 podStartE2EDuration="4.236137501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.234319601 +0000 UTC m=+1965.976250277" watchObservedRunningTime="2026-10-29 22:15:04.236137501 +0000 UTC m=+1965.978068175" Oct 29 22:15:04.282991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.282937 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" podStartSLOduration=4.282921017 podStartE2EDuration="4.282921017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.279440092 +0000 UTC m=+1966.021370765" watchObservedRunningTime="2026-10-29 22:15:04.282921017 +0000 UTC m=+1966.024851692" Oct 29 22:15:04.401317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.401237 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" podStartSLOduration=4.401212022 podStartE2EDuration="4.401212022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.368434939 +0000 UTC m=+1966.110365613" watchObservedRunningTime="2026-10-29 22:15:04.401212022 +0000 UTC m=+1966.143142697" Oct 29 22:15:04.420497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.420456 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25fed8d4-2429-48a0-bbf6-e70608e178d7" path="/var/lib/kubelet/pods/25fed8d4-2429-48a0-bbf6-e70608e178d7/volumes" Oct 29 22:15:04.426855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.426790 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" podStartSLOduration=4.426770801 podStartE2EDuration="4.426770801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.420784695 +0000 UTC m=+1966.162715370" watchObservedRunningTime="2026-10-29 22:15:04.426770801 +0000 UTC m=+1966.168701475" Oct 29 22:15:04.511660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.511610 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" podStartSLOduration=4.511593704 podStartE2EDuration="4.511593704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.506779227 +0000 UTC m=+1966.248709899" watchObservedRunningTime="2026-10-29 22:15:04.511593704 +0000 UTC m=+1966.253524378" Oct 29 22:15:04.551616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:04.551571 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" podStartSLOduration=4.55155845 podStartE2EDuration="4.55155845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.550063323 +0000 UTC m=+1966.291993996" watchObservedRunningTime="2026-10-29 22:15:04.55155845 +0000 UTC m=+1966.293489157" Oct 29 22:15:05.091424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.091336 4543 generic.go:356] "Generic (PLEG): container finished" podID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerID="03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a" exitCode=60 Oct 29 22:15:05.091866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.091430 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerDied","Data":"03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a"} Oct 29 22:15:05.091866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.091528 4543 scope.go:122] "RemoveContainer" containerID="4b652b576bd625956ca0b15a48f04a8ce29ca9474e341e5d042f8cd547c8ffda" Oct 29 22:15:05.091866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.091802 4543 scope.go:122] "RemoveContainer" containerID="03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a" Oct 29 22:15:05.092292 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.092266 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz_odh-ai-gateway-infra(a71d77a0-9414-40e6-8237-5cb0d7697a41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" Oct 29 22:15:05.093750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.093729 4543 generic.go:356] "Generic (PLEG): container finished" podID="647f181c-5a88-44f7-886f-c255f46319c4" containerID="6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df" exitCode=60 Oct 29 22:15:05.093841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.093799 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerDied","Data":"6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df"} Oct 29 22:15:05.094178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.094159 4543 scope.go:122] "RemoveContainer" containerID="6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df" Oct 29 22:15:05.094586 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.094563 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l_odh-ai-gateway-infra(647f181c-5a88-44f7-886f-c255f46319c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" podUID="647f181c-5a88-44f7-886f-c255f46319c4" Oct 29 22:15:05.096273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.096253 4543 generic.go:356] "Generic (PLEG): container finished" podID="64b12988-8477-44bb-a903-d780202fb9e0" containerID="b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a" exitCode=60 Oct 29 22:15:05.096352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.096332 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerDied","Data":"b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a"} Oct 29 22:15:05.096585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.096570 4543 scope.go:122] "RemoveContainer" containerID="b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a" Oct 29 22:15:05.096900 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.096877 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg_odh-ai-gateway-infra(64b12988-8477-44bb-a903-d780202fb9e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" podUID="64b12988-8477-44bb-a903-d780202fb9e0" Oct 29 22:15:05.098495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.098475 4543 generic.go:356] "Generic (PLEG): container finished" podID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerID="4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e" exitCode=60 Oct 29 22:15:05.098571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.098541 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerDied","Data":"4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e"} Oct 29 22:15:05.098825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.098807 4543 scope.go:122] "RemoveContainer" containerID="4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e" Oct 29 22:15:05.099198 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.099171 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc_odh-ai-gateway-infra(ec53711c-2418-4464-bd4a-2ca258fe10e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" Oct 29 22:15:05.100718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.100696 4543 generic.go:356] "Generic (PLEG): container finished" podID="71348e81-9a5d-4738-ad0c-561ea7442536" containerID="1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9" exitCode=60 Oct 29 22:15:05.100793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.100756 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerDied","Data":"1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9"} Oct 29 22:15:05.101057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.101039 4543 scope.go:122] "RemoveContainer" containerID="1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9" Oct 29 22:15:05.101384 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.101363 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69_odh-ai-gateway-infra(71348e81-9a5d-4738-ad0c-561ea7442536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" Oct 29 22:15:05.102969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.102949 4543 generic.go:356] "Generic (PLEG): container finished" podID="da8185d5-4592-4761-b078-7cd63d635020" containerID="0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954" exitCode=60 Oct 29 22:15:05.103053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.103028 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerDied","Data":"0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954"} Oct 29 22:15:05.103238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.103221 4543 scope.go:122] "RemoveContainer" containerID="0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954" Oct 29 22:15:05.103509 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.103482 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g_odh-ai-gateway-infra(da8185d5-4592-4761-b078-7cd63d635020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" podUID="da8185d5-4592-4761-b078-7cd63d635020" Oct 29 22:15:05.105181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.105160 4543 generic.go:356] "Generic (PLEG): container finished" podID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerID="c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f" exitCode=60 Oct 29 22:15:05.105283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.105255 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerDied","Data":"c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f"} Oct 29 22:15:05.105513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.105499 4543 scope.go:122] "RemoveContainer" containerID="c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f" Oct 29 22:15:05.105791 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.105772 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp_odh-ai-gateway-infra(b3d4c4a3-deda-4ad9-a481-d7bf244440cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" Oct 29 22:15:05.107320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.107260 4543 generic.go:356] "Generic (PLEG): container finished" podID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerID="032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507" exitCode=60 Oct 29 22:15:05.107383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.107332 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerDied","Data":"032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507"} Oct 29 22:15:05.107431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.107415 4543 scope.go:122] "RemoveContainer" containerID="814b716b9721027cf479959dd7be1a1e5e2c63aa195d9f96a4fa8a71972951a4" Oct 29 22:15:05.107597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.107576 4543 scope.go:122] "RemoveContainer" containerID="032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507" Oct 29 22:15:05.107895 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.107871 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s_odh-ai-gateway-infra(77ce399b-2b7f-4c84-8362-89a4efcfca34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" Oct 29 22:15:05.109649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.109630 4543 generic.go:356] "Generic (PLEG): container finished" podID="04542cbe-3637-4338-a594-49c7406422bf" containerID="31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df" exitCode=60 Oct 29 22:15:05.109795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.109651 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerDied","Data":"31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df"} Oct 29 22:15:05.109911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.109893 4543 scope.go:122] "RemoveContainer" containerID="31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df" Oct 29 22:15:05.110225 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.110201 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn_odh-ai-gateway-infra(04542cbe-3637-4338-a594-49c7406422bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" podUID="04542cbe-3637-4338-a594-49c7406422bf" Oct 29 22:15:05.111696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.111673 4543 generic.go:356] "Generic (PLEG): container finished" podID="e2984b7c-87fe-42ce-a046-81066764caad" containerID="9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57" exitCode=60 Oct 29 22:15:05.111789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.111736 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerDied","Data":"9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57"} Oct 29 22:15:05.111977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.111961 4543 scope.go:122] "RemoveContainer" containerID="9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57" Oct 29 22:15:05.112247 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.112223 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq_odh-ai-gateway-infra(e2984b7c-87fe-42ce-a046-81066764caad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" podUID="e2984b7c-87fe-42ce-a046-81066764caad" Oct 29 22:15:05.114144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.114119 4543 generic.go:356] "Generic (PLEG): container finished" podID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerID="3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992" exitCode=60 Oct 29 22:15:05.114236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.114143 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerDied","Data":"3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992"} Oct 29 22:15:05.114402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.114385 4543 scope.go:122] "RemoveContainer" containerID="3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992" Oct 29 22:15:05.114629 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:05.114610 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v_odh-ai-gateway-infra(db2f1188-e008-45a9-b06c-6bcb05625eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" Oct 29 22:15:05.120976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.120957 4543 scope.go:122] "RemoveContainer" containerID="6e26cec204df6e3d37b58ca48b6d38d2941bfa0e36bca6892d7195e6fc1793d9" Oct 29 22:15:05.133712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.133695 4543 scope.go:122] "RemoveContainer" containerID="5bd359f5bc7e847f83a3d17dfc9bb6e82713567ad538790d65728f0718688e22" Oct 29 22:15:05.145817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.145796 4543 scope.go:122] "RemoveContainer" containerID="4e6655356456672bcde531ed1489fb5da163e0387d0b7445b3ec1cb117fc769c" Oct 29 22:15:05.157892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.157873 4543 scope.go:122] "RemoveContainer" containerID="d046f30733f1f4b7f66571af422d532706d0591ce396952dbc5a223276c1cfd8" Oct 29 22:15:05.175911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.175891 4543 scope.go:122] "RemoveContainer" containerID="7a4459aa59c38c80b9a4ec5efd5d870ec789444d84d85dd7f57c467029476298" Oct 29 22:15:05.189172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.189155 4543 scope.go:122] "RemoveContainer" containerID="25b2c14b521423ab98ea280396ce969e12cbbaed84dfe6e7c473981fa36f6641" Oct 29 22:15:05.201516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.201498 4543 scope.go:122] "RemoveContainer" containerID="8d72039b7b21d1f64b140c28415180d95b577b02cbb1f43bbcec1ff4639b732f" Oct 29 22:15:05.213827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.213806 4543 scope.go:122] "RemoveContainer" containerID="acf1da4ddb3cf10651623101f3ffa56e23816fadb15121dd04e5f8c6b9450aed" Oct 29 22:15:05.227971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:05.227951 4543 scope.go:122] "RemoveContainer" containerID="243c5c339923b24795ea678209a67cf9f006253a61eea91e691913833edcdc20" Oct 29 22:15:06.036727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.036693 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz"] Oct 29 22:15:06.049957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.049928 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05283aed-dddnz"] Oct 29 22:15:06.127590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.127565 4543 scope.go:122] "RemoveContainer" containerID="03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a" Oct 29 22:15:06.128031 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.127929 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz_odh-ai-gateway-infra(a71d77a0-9414-40e6-8237-5cb0d7697a41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" Oct 29 22:15:06.129493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.129470 4543 scope.go:122] "RemoveContainer" containerID="6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df" Oct 29 22:15:06.129749 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.129721 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l_odh-ai-gateway-infra(647f181c-5a88-44f7-886f-c255f46319c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" podUID="647f181c-5a88-44f7-886f-c255f46319c4" Oct 29 22:15:06.131162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.131146 4543 scope.go:122] "RemoveContainer" containerID="b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a" Oct 29 22:15:06.131384 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.131357 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg_odh-ai-gateway-infra(64b12988-8477-44bb-a903-d780202fb9e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" podUID="64b12988-8477-44bb-a903-d780202fb9e0" Oct 29 22:15:06.132999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.132969 4543 scope.go:122] "RemoveContainer" containerID="4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e" Oct 29 22:15:06.133273 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.133250 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc_odh-ai-gateway-infra(ec53711c-2418-4464-bd4a-2ca258fe10e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" Oct 29 22:15:06.134774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.134758 4543 scope.go:122] "RemoveContainer" containerID="1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9" Oct 29 22:15:06.134982 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.134965 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69_odh-ai-gateway-infra(71348e81-9a5d-4738-ad0c-561ea7442536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" Oct 29 22:15:06.136388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.136370 4543 scope.go:122] "RemoveContainer" containerID="0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954" Oct 29 22:15:06.136626 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.136598 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g_odh-ai-gateway-infra(da8185d5-4592-4761-b078-7cd63d635020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" podUID="da8185d5-4592-4761-b078-7cd63d635020" Oct 29 22:15:06.138239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.138220 4543 scope.go:122] "RemoveContainer" containerID="c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f" Oct 29 22:15:06.138442 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.138424 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp_odh-ai-gateway-infra(b3d4c4a3-deda-4ad9-a481-d7bf244440cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" Oct 29 22:15:06.139810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.139781 4543 scope.go:122] "RemoveContainer" containerID="032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507" Oct 29 22:15:06.140010 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.139988 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s_odh-ai-gateway-infra(77ce399b-2b7f-4c84-8362-89a4efcfca34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" Oct 29 22:15:06.141342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.141323 4543 scope.go:122] "RemoveContainer" containerID="31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df" Oct 29 22:15:06.141564 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.141545 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn_odh-ai-gateway-infra(04542cbe-3637-4338-a594-49c7406422bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" podUID="04542cbe-3637-4338-a594-49c7406422bf" Oct 29 22:15:06.143112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.143095 4543 scope.go:122] "RemoveContainer" containerID="9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57" Oct 29 22:15:06.143347 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.143319 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq_odh-ai-gateway-infra(e2984b7c-87fe-42ce-a046-81066764caad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" podUID="e2984b7c-87fe-42ce-a046-81066764caad" Oct 29 22:15:06.144694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.144679 4543 scope.go:122] "RemoveContainer" containerID="3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992" Oct 29 22:15:06.144893 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:06.144876 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v_odh-ai-gateway-infra(db2f1188-e008-45a9-b06c-6bcb05625eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" Oct 29 22:15:06.388799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:06.388722 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b831047-e15d-424b-9703-58b650108c17" path="/var/lib/kubelet/pods/1b831047-e15d-424b-9703-58b650108c17/volumes" Oct 29 22:15:15.379043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:15.378132 4543 scope.go:122] "RemoveContainer" containerID="1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9" Oct 29 22:15:15.379043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:15.378382 4543 scope.go:122] "RemoveContainer" containerID="4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e" Oct 29 22:15:15.379043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:15.378967 4543 scope.go:122] "RemoveContainer" containerID="31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df" Oct 29 22:15:16.247722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.247687 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerStarted","Data":"4008c05d8bb4f44719a6b462e1ea638038b205e9d1d68d06affccaec104c369a"} Oct 29 22:15:16.249763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.249738 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerStarted","Data":"1f0e39d70db84715a8eefd7410ce74691169f16e5a5286c8e907a2ce29fa2c6f"} Oct 29 22:15:16.251818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.251796 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerStarted","Data":"01465e9c94f857620c3393c9f523bee976a76b7bfda561b2b31fd1ba29ab3cc5"} Oct 29 22:15:16.377766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.377742 4543 scope.go:122] "RemoveContainer" containerID="9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57" Oct 29 22:15:16.377938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.377903 4543 scope.go:122] "RemoveContainer" containerID="b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a" Oct 29 22:15:16.378048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.378029 4543 scope.go:122] "RemoveContainer" containerID="0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954" Oct 29 22:15:16.378572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.378548 4543 scope.go:122] "RemoveContainer" containerID="032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507" Oct 29 22:15:16.378876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:16.378857 4543 scope.go:122] "RemoveContainer" containerID="3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992" Oct 29 22:15:17.263485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.263461 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerStarted","Data":"508757bdd00502a15ecb3a7ffa2807b293eb93e856ace6a2b11c7561b71408d0"} Oct 29 22:15:17.265802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.265779 4543 generic.go:356] "Generic (PLEG): container finished" podID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerID="4008c05d8bb4f44719a6b462e1ea638038b205e9d1d68d06affccaec104c369a" exitCode=60 Oct 29 22:15:17.265962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.265867 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerDied","Data":"4008c05d8bb4f44719a6b462e1ea638038b205e9d1d68d06affccaec104c369a"} Oct 29 22:15:17.265962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.265921 4543 scope.go:122] "RemoveContainer" containerID="4736adb451121a788227a66edacbd8549efa93e776b85279428475a65154a75e" Oct 29 22:15:17.266292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.266271 4543 scope.go:122] "RemoveContainer" containerID="4008c05d8bb4f44719a6b462e1ea638038b205e9d1d68d06affccaec104c369a" Oct 29 22:15:17.266612 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:17.266588 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc_odh-ai-gateway-infra(ec53711c-2418-4464-bd4a-2ca258fe10e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" Oct 29 22:15:17.268060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.268041 4543 generic.go:356] "Generic (PLEG): container finished" podID="71348e81-9a5d-4738-ad0c-561ea7442536" containerID="1f0e39d70db84715a8eefd7410ce74691169f16e5a5286c8e907a2ce29fa2c6f" exitCode=60 Oct 29 22:15:17.268175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.268126 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerDied","Data":"1f0e39d70db84715a8eefd7410ce74691169f16e5a5286c8e907a2ce29fa2c6f"} Oct 29 22:15:17.268486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.268468 4543 scope.go:122] "RemoveContainer" containerID="1f0e39d70db84715a8eefd7410ce74691169f16e5a5286c8e907a2ce29fa2c6f" Oct 29 22:15:17.268779 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:17.268760 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69_odh-ai-gateway-infra(71348e81-9a5d-4738-ad0c-561ea7442536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" Oct 29 22:15:17.270285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.270262 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerStarted","Data":"99175af287763005d217d9b64ef48a1d7ed063847c5b26560e760c9039aad82d"} Oct 29 22:15:17.272376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.272352 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerStarted","Data":"765345957b408ab3502bf7a1700e7feb4fd560e02840f237bcee1dd0ef5a2d5b"} Oct 29 22:15:17.274412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.274394 4543 generic.go:356] "Generic (PLEG): container finished" podID="04542cbe-3637-4338-a594-49c7406422bf" containerID="01465e9c94f857620c3393c9f523bee976a76b7bfda561b2b31fd1ba29ab3cc5" exitCode=60 Oct 29 22:15:17.274491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.274455 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerDied","Data":"01465e9c94f857620c3393c9f523bee976a76b7bfda561b2b31fd1ba29ab3cc5"} Oct 29 22:15:17.274745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.274728 4543 scope.go:122] "RemoveContainer" containerID="01465e9c94f857620c3393c9f523bee976a76b7bfda561b2b31fd1ba29ab3cc5" Oct 29 22:15:17.274964 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:17.274937 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn_odh-ai-gateway-infra(04542cbe-3637-4338-a594-49c7406422bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" podUID="04542cbe-3637-4338-a594-49c7406422bf" Oct 29 22:15:17.276826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.276798 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerStarted","Data":"008b94aa12f65d4222a4cb5e5a1fe51ca071f097af5419c438072ca93450cc37"} Oct 29 22:15:17.278632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.278613 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerStarted","Data":"d1f4e3f151591c9c740a1f2e5c6ae39db1c7910f7309cbe77bd2d5c50e6ec369"} Oct 29 22:15:17.282003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.281985 4543 scope.go:122] "RemoveContainer" containerID="1565d7d302ed3b760cfe4175a870bca12172a36e4812eed75516511fde90dde9" Oct 29 22:15:17.297429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.297396 4543 scope.go:122] "RemoveContainer" containerID="31d229ffd530eff406ab546627a69555cfe2cde23a86fdd84ac531015e1c29df" Oct 29 22:15:17.308150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.308131 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:17.351660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.351635 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:17.377841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.377820 4543 scope.go:122] "RemoveContainer" containerID="c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f" Oct 29 22:15:17.378027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.378011 4543 scope.go:122] "RemoveContainer" containerID="6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df" Oct 29 22:15:17.394977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:17.394930 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:18.291719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.291691 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerStarted","Data":"e275ca685e2e224463d5cd8bd240127e62f659ee6e6e561367de803b62a8f337"} Oct 29 22:15:18.293811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.293790 4543 generic.go:356] "Generic (PLEG): container finished" podID="64b12988-8477-44bb-a903-d780202fb9e0" containerID="508757bdd00502a15ecb3a7ffa2807b293eb93e856ace6a2b11c7561b71408d0" exitCode=60 Oct 29 22:15:18.293957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.293864 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerDied","Data":"508757bdd00502a15ecb3a7ffa2807b293eb93e856ace6a2b11c7561b71408d0"} Oct 29 22:15:18.293957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.293923 4543 scope.go:122] "RemoveContainer" containerID="b9464f0f67eabc712e2eeef2206336482ac3caee93272b8c4090b15562c98a9a" Oct 29 22:15:18.294213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.294199 4543 scope.go:122] "RemoveContainer" containerID="508757bdd00502a15ecb3a7ffa2807b293eb93e856ace6a2b11c7561b71408d0" Oct 29 22:15:18.294534 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:18.294507 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg_odh-ai-gateway-infra(64b12988-8477-44bb-a903-d780202fb9e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" podUID="64b12988-8477-44bb-a903-d780202fb9e0" Oct 29 22:15:18.300203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.300177 4543 generic.go:356] "Generic (PLEG): container finished" podID="da8185d5-4592-4761-b078-7cd63d635020" containerID="99175af287763005d217d9b64ef48a1d7ed063847c5b26560e760c9039aad82d" exitCode=60 Oct 29 22:15:18.300360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.300246 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerDied","Data":"99175af287763005d217d9b64ef48a1d7ed063847c5b26560e760c9039aad82d"} Oct 29 22:15:18.300574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.300558 4543 scope.go:122] "RemoveContainer" containerID="99175af287763005d217d9b64ef48a1d7ed063847c5b26560e760c9039aad82d" Oct 29 22:15:18.300941 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:18.300852 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g_odh-ai-gateway-infra(da8185d5-4592-4761-b078-7cd63d635020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" podUID="da8185d5-4592-4761-b078-7cd63d635020" Oct 29 22:15:18.303090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.303049 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerStarted","Data":"9f3cc92d8841b0111188906716070d483e93b4695dcdacc1096934dfb76b76a3"} Oct 29 22:15:18.311190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.311165 4543 generic.go:356] "Generic (PLEG): container finished" podID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerID="765345957b408ab3502bf7a1700e7feb4fd560e02840f237bcee1dd0ef5a2d5b" exitCode=60 Oct 29 22:15:18.311362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.311342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerDied","Data":"765345957b408ab3502bf7a1700e7feb4fd560e02840f237bcee1dd0ef5a2d5b"} Oct 29 22:15:18.311652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.311634 4543 scope.go:122] "RemoveContainer" containerID="765345957b408ab3502bf7a1700e7feb4fd560e02840f237bcee1dd0ef5a2d5b" Oct 29 22:15:18.311911 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:18.311888 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s_odh-ai-gateway-infra(77ce399b-2b7f-4c84-8362-89a4efcfca34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" Oct 29 22:15:18.316546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.316528 4543 scope.go:122] "RemoveContainer" containerID="0a8c8c407dc7ab03ea152d61d549cc2b9e89d5a9294c53d352a3580928680954" Oct 29 22:15:18.318562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.318535 4543 generic.go:356] "Generic (PLEG): container finished" podID="e2984b7c-87fe-42ce-a046-81066764caad" containerID="008b94aa12f65d4222a4cb5e5a1fe51ca071f097af5419c438072ca93450cc37" exitCode=60 Oct 29 22:15:18.318661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.318623 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerDied","Data":"008b94aa12f65d4222a4cb5e5a1fe51ca071f097af5419c438072ca93450cc37"} Oct 29 22:15:18.318991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.318972 4543 scope.go:122] "RemoveContainer" containerID="008b94aa12f65d4222a4cb5e5a1fe51ca071f097af5419c438072ca93450cc37" Oct 29 22:15:18.319362 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:18.319339 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq_odh-ai-gateway-infra(e2984b7c-87fe-42ce-a046-81066764caad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" podUID="e2984b7c-87fe-42ce-a046-81066764caad" Oct 29 22:15:18.322420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.322367 4543 generic.go:356] "Generic (PLEG): container finished" podID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerID="d1f4e3f151591c9c740a1f2e5c6ae39db1c7910f7309cbe77bd2d5c50e6ec369" exitCode=60 Oct 29 22:15:18.322653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.322599 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerDied","Data":"d1f4e3f151591c9c740a1f2e5c6ae39db1c7910f7309cbe77bd2d5c50e6ec369"} Oct 29 22:15:18.323113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.323044 4543 scope.go:122] "RemoveContainer" containerID="d1f4e3f151591c9c740a1f2e5c6ae39db1c7910f7309cbe77bd2d5c50e6ec369" Oct 29 22:15:18.323396 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:18.323376 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v_odh-ai-gateway-infra(db2f1188-e008-45a9-b06c-6bcb05625eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" Oct 29 22:15:18.332016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.331937 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:18.351704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.351680 4543 scope.go:122] "RemoveContainer" containerID="032ab4a5a23fde43916b0593d164b41b33068d5ee4d0d4c96cd27fc667ce0507" Oct 29 22:15:18.371910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.371891 4543 scope.go:122] "RemoveContainer" containerID="9523f8514a62384995a954868dc878151750ceac7ed7d09cabe3be8e6b0bfc57" Oct 29 22:15:18.376196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.376177 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:18.379905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.379128 4543 scope.go:122] "RemoveContainer" containerID="03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a" Oct 29 22:15:18.445033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.444940 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:18.449515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.449496 4543 scope.go:122] "RemoveContainer" containerID="3105755a9bfd15af05245b946bd6e5f9a34bda1a4da738676bfb15f9ba132992" Oct 29 22:15:18.601165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.600892 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:18.638317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.638288 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:18.668323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.668293 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:18.672478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.672453 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:18.676552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.676533 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:18.699809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.699785 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\" (UniqueName: \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\") pod \"ec53711c-2418-4464-bd4a-2ca258fe10e3\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " Oct 29 22:15:18.699955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.699856 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\") pod \"ec53711c-2418-4464-bd4a-2ca258fe10e3\" (UID: \"ec53711c-2418-4464-bd4a-2ca258fe10e3\") " Oct 29 22:15:18.700301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.700278 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca" pod "ec53711c-2418-4464-bd4a-2ca258fe10e3" (UID: "ec53711c-2418-4464-bd4a-2ca258fe10e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:18.700662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.700638 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec53711c-2418-4464-bd4a-2ca258fe10e3-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:18.702589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.702566 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c" pod "ec53711c-2418-4464-bd4a-2ca258fe10e3" (UID: "ec53711c-2418-4464-bd4a-2ca258fe10e3"). InnerVolumeSpecName "kube-api-access-rj98c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:18.802037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.801953 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\" (UniqueName: \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\") pod \"04542cbe-3637-4338-a594-49c7406422bf\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " Oct 29 22:15:18.802239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.802133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\") pod \"04542cbe-3637-4338-a594-49c7406422bf\" (UID: \"04542cbe-3637-4338-a594-49c7406422bf\") " Oct 29 22:15:18.802304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.802277 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\") pod \"71348e81-9a5d-4738-ad0c-561ea7442536\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " Oct 29 22:15:18.802427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.802405 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\" (UniqueName: \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\") pod \"71348e81-9a5d-4738-ad0c-561ea7442536\" (UID: \"71348e81-9a5d-4738-ad0c-561ea7442536\") " Oct 29 22:15:18.802574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.802486 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca" pod "04542cbe-3637-4338-a594-49c7406422bf" (UID: "04542cbe-3637-4338-a594-49c7406422bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:18.802738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.802719 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca" pod "71348e81-9a5d-4738-ad0c-561ea7442536" (UID: "71348e81-9a5d-4738-ad0c-561ea7442536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:18.803066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.803049 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04542cbe-3637-4338-a594-49c7406422bf-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:18.803148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.803069 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj98c\" (UniqueName: \"kubernetes.io/projected/ec53711c-2418-4464-bd4a-2ca258fe10e3-kube-api-access-rj98c\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:18.803148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.803102 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71348e81-9a5d-4738-ad0c-561ea7442536-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:18.804572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.804549 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg" pod "04542cbe-3637-4338-a594-49c7406422bf" (UID: "04542cbe-3637-4338-a594-49c7406422bf"). InnerVolumeSpecName "kube-api-access-pcvmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:18.805114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.805068 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq" pod "71348e81-9a5d-4738-ad0c-561ea7442536" (UID: "71348e81-9a5d-4738-ad0c-561ea7442536"). InnerVolumeSpecName "kube-api-access-8xmgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:18.904972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.904937 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmgq\" (UniqueName: \"kubernetes.io/projected/71348e81-9a5d-4738-ad0c-561ea7442536-kube-api-access-8xmgq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:18.905151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:18.904975 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvmg\" (UniqueName: \"kubernetes.io/projected/04542cbe-3637-4338-a594-49c7406422bf-kube-api-access-pcvmg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.337065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.337036 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" Oct 29 22:15:19.337581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.337030 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn" event={"ID":"04542cbe-3637-4338-a594-49c7406422bf","Type":"ContainerDied","Data":"3eedf44b141a624142621f1dbcc83646447562fbff8e4cd5e51c3ee8065d10c9"} Oct 29 22:15:19.337581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.337256 4543 scope.go:122] "RemoveContainer" containerID="01465e9c94f857620c3393c9f523bee976a76b7bfda561b2b31fd1ba29ab3cc5" Oct 29 22:15:19.345791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.345761 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerStarted","Data":"1b1b2ace8dab2e823d40c09ce474f894d964d39cceb8996edbc4da7d10b3075a"} Oct 29 22:15:19.353036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.353005 4543 generic.go:356] "Generic (PLEG): container finished" podID="647f181c-5a88-44f7-886f-c255f46319c4" containerID="e275ca685e2e224463d5cd8bd240127e62f659ee6e6e561367de803b62a8f337" exitCode=60 Oct 29 22:15:19.353183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.353108 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerDied","Data":"e275ca685e2e224463d5cd8bd240127e62f659ee6e6e561367de803b62a8f337"} Oct 29 22:15:19.353514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.353494 4543 scope.go:122] "RemoveContainer" containerID="e275ca685e2e224463d5cd8bd240127e62f659ee6e6e561367de803b62a8f337" Oct 29 22:15:19.353849 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:19.353823 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l_odh-ai-gateway-infra(647f181c-5a88-44f7-886f-c255f46319c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" podUID="647f181c-5a88-44f7-886f-c255f46319c4" Oct 29 22:15:19.364237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.363936 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:19.364480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.364456 4543 scope.go:122] "RemoveContainer" containerID="6b0304e2262e69f31244cb72af18365597bbc896d4fa5cf1d7d28e20fa6696df" Oct 29 22:15:19.397108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.395293 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" event={"ID":"ec53711c-2418-4464-bd4a-2ca258fe10e3","Type":"ContainerDied","Data":"0bccd17dd9f979f1760742ed8724be14a57fa756d5a22fcc46ed1c29f1638365"} Oct 29 22:15:19.397108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.395475 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc" Oct 29 22:15:19.428878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.428692 4543 scope.go:122] "RemoveContainer" containerID="4008c05d8bb4f44719a6b462e1ea638038b205e9d1d68d06affccaec104c369a" Oct 29 22:15:19.430918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.430891 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" event={"ID":"71348e81-9a5d-4738-ad0c-561ea7442536","Type":"ContainerDied","Data":"1ccdce50e9cbbcfbaa768230d7c27c2fa5d3760e5a035102310f7c4580a32d81"} Oct 29 22:15:19.431317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.431114 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69" Oct 29 22:15:19.446721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.441585 4543 generic.go:356] "Generic (PLEG): container finished" podID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerID="9f3cc92d8841b0111188906716070d483e93b4695dcdacc1096934dfb76b76a3" exitCode=60 Oct 29 22:15:19.446721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.441697 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerDied","Data":"9f3cc92d8841b0111188906716070d483e93b4695dcdacc1096934dfb76b76a3"} Oct 29 22:15:19.446721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.442186 4543 scope.go:122] "RemoveContainer" containerID="9f3cc92d8841b0111188906716070d483e93b4695dcdacc1096934dfb76b76a3" Oct 29 22:15:19.446721 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:19.442684 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp_odh-ai-gateway-infra(b3d4c4a3-deda-4ad9-a481-d7bf244440cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" Oct 29 22:15:19.448125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.448095 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:19.459692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.459595 4543 scope.go:122] "RemoveContainer" containerID="1f0e39d70db84715a8eefd7410ce74691169f16e5a5286c8e907a2ce29fa2c6f" Oct 29 22:15:19.530265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.530227 4543 scope.go:122] "RemoveContainer" containerID="c0b922d083136124aab597cd238c2ce4110e4694017428a50efb990277bf764f" Oct 29 22:15:19.703026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.702323 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:19.709541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.709516 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888535-dgm69"] Oct 29 22:15:19.732458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.732260 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:19.755183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.755149 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:19.761485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.761458 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888535-k9hcn"] Oct 29 22:15:19.774859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.774834 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:19.783731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.783708 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:19.789620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.789580 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:19.799091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.796643 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:19.833299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.833266 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\" (UniqueName: \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\") pod \"e2984b7c-87fe-42ce-a046-81066764caad\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " Oct 29 22:15:19.833436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.833406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\" (UniqueName: \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\") pod \"db2f1188-e008-45a9-b06c-6bcb05625eec\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " Oct 29 22:15:19.833543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.833524 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\") pod \"db2f1188-e008-45a9-b06c-6bcb05625eec\" (UID: \"db2f1188-e008-45a9-b06c-6bcb05625eec\") " Oct 29 22:15:19.833719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.833699 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\") pod \"e2984b7c-87fe-42ce-a046-81066764caad\" (UID: \"e2984b7c-87fe-42ce-a046-81066764caad\") " Oct 29 22:15:19.834012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.833878 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca" pod "db2f1188-e008-45a9-b06c-6bcb05625eec" (UID: "db2f1188-e008-45a9-b06c-6bcb05625eec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.834413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.834333 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca" pod "e2984b7c-87fe-42ce-a046-81066764caad" (UID: "e2984b7c-87fe-42ce-a046-81066764caad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.834701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.834681 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db2f1188-e008-45a9-b06c-6bcb05625eec-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.834783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.834708 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2984b7c-87fe-42ce-a046-81066764caad-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.835994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.835970 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96" pod "db2f1188-e008-45a9-b06c-6bcb05625eec" (UID: "db2f1188-e008-45a9-b06c-6bcb05625eec"). InnerVolumeSpecName "kube-api-access-72f96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.836219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.836200 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz" pod "e2984b7c-87fe-42ce-a046-81066764caad" (UID: "e2984b7c-87fe-42ce-a046-81066764caad"). InnerVolumeSpecName "kube-api-access-56gpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.840147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.840128 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:19.845174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.845130 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888535-b9xcc"] Oct 29 22:15:19.935413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935387 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\" (UniqueName: \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\") pod \"77ce399b-2b7f-4c84-8362-89a4efcfca34\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " Oct 29 22:15:19.935534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935422 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\" (UniqueName: \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\") pod \"da8185d5-4592-4761-b078-7cd63d635020\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " Oct 29 22:15:19.935534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935445 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\" (UniqueName: \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\") pod \"64b12988-8477-44bb-a903-d780202fb9e0\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " Oct 29 22:15:19.935534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935508 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\") pod \"64b12988-8477-44bb-a903-d780202fb9e0\" (UID: \"64b12988-8477-44bb-a903-d780202fb9e0\") " Oct 29 22:15:19.935724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\") pod \"77ce399b-2b7f-4c84-8362-89a4efcfca34\" (UID: \"77ce399b-2b7f-4c84-8362-89a4efcfca34\") " Oct 29 22:15:19.936058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935871 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca" pod "64b12988-8477-44bb-a903-d780202fb9e0" (UID: "64b12988-8477-44bb-a903-d780202fb9e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.936058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.935947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\") pod \"da8185d5-4592-4761-b078-7cd63d635020\" (UID: \"da8185d5-4592-4761-b078-7cd63d635020\") " Oct 29 22:15:19.936246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.936206 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca" pod "77ce399b-2b7f-4c84-8362-89a4efcfca34" (UID: "77ce399b-2b7f-4c84-8362-89a4efcfca34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.936419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.936393 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca" pod "da8185d5-4592-4761-b078-7cd63d635020" (UID: "da8185d5-4592-4761-b078-7cd63d635020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.937031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.937009 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8185d5-4592-4761-b078-7cd63d635020-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.937127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.937039 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gpz\" (UniqueName: \"kubernetes.io/projected/e2984b7c-87fe-42ce-a046-81066764caad-kube-api-access-56gpz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.937127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.937055 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72f96\" (UniqueName: \"kubernetes.io/projected/db2f1188-e008-45a9-b06c-6bcb05625eec-kube-api-access-72f96\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.937127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.937069 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b12988-8477-44bb-a903-d780202fb9e0-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.937127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.937101 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ce399b-2b7f-4c84-8362-89a4efcfca34-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.938178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.938133 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6" pod "77ce399b-2b7f-4c84-8362-89a4efcfca34" (UID: "77ce399b-2b7f-4c84-8362-89a4efcfca34"). InnerVolumeSpecName "kube-api-access-8zdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.938764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.938742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw" pod "64b12988-8477-44bb-a903-d780202fb9e0" (UID: "64b12988-8477-44bb-a903-d780202fb9e0"). InnerVolumeSpecName "kube-api-access-6vvdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.938908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:19.938842 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp" pod "da8185d5-4592-4761-b078-7cd63d635020" (UID: "da8185d5-4592-4761-b078-7cd63d635020"). InnerVolumeSpecName "kube-api-access-hsjkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:20.039120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.039067 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdf6\" (UniqueName: \"kubernetes.io/projected/77ce399b-2b7f-4c84-8362-89a4efcfca34-kube-api-access-8zdf6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.039369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.039336 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsjkp\" (UniqueName: \"kubernetes.io/projected/da8185d5-4592-4761-b078-7cd63d635020-kube-api-access-hsjkp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.039369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.039366 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vvdw\" (UniqueName: \"kubernetes.io/projected/64b12988-8477-44bb-a903-d780202fb9e0-kube-api-access-6vvdw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.387613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.387580 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04542cbe-3637-4338-a594-49c7406422bf" path="/var/lib/kubelet/pods/04542cbe-3637-4338-a594-49c7406422bf/volumes" Oct 29 22:15:20.388070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.388054 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" path="/var/lib/kubelet/pods/71348e81-9a5d-4738-ad0c-561ea7442536/volumes" Oct 29 22:15:20.388509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.388495 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" path="/var/lib/kubelet/pods/ec53711c-2418-4464-bd4a-2ca258fe10e3/volumes" Oct 29 22:15:20.581611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.581576 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:20.602938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.602900 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" event={"ID":"da8185d5-4592-4761-b078-7cd63d635020","Type":"ContainerDied","Data":"7ad47615fbfb18d481545f6a4a3e2a46724ae1f84807936ffb94a38bf923e353"} Oct 29 22:15:20.614907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.614881 4543 scope.go:122] "RemoveContainer" containerID="99175af287763005d217d9b64ef48a1d7ed063847c5b26560e760c9039aad82d" Oct 29 22:15:20.615219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.603293 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g" Oct 29 22:15:20.649423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.643456 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" event={"ID":"77ce399b-2b7f-4c84-8362-89a4efcfca34","Type":"ContainerDied","Data":"e084384671e606f6e3246b982c3c62f14467b7ff11508bae0d29d62be22184c9"} Oct 29 22:15:20.649423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.643680 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s" Oct 29 22:15:20.668475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.659352 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" event={"ID":"e2984b7c-87fe-42ce-a046-81066764caad","Type":"ContainerDied","Data":"c22e83a0768e4f6343eae565e2c67c4903bcaa0a7f2c8a1ce08aa3b526978b43"} Oct 29 22:15:20.668475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.659477 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq" Oct 29 22:15:20.695679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.679565 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" Oct 29 22:15:20.695679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.679561 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v" event={"ID":"db2f1188-e008-45a9-b06c-6bcb05625eec","Type":"ContainerDied","Data":"61692630de3a88b55fe2dc84fdada9bd678c3acfbd2cc3dae38c236132f7fe2b"} Oct 29 22:15:20.695679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.682152 4543 generic.go:356] "Generic (PLEG): container finished" podID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerID="1b1b2ace8dab2e823d40c09ce474f894d964d39cceb8996edbc4da7d10b3075a" exitCode=60 Oct 29 22:15:20.695679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.682257 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerDied","Data":"1b1b2ace8dab2e823d40c09ce474f894d964d39cceb8996edbc4da7d10b3075a"} Oct 29 22:15:20.710682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.707032 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" event={"ID":"64b12988-8477-44bb-a903-d780202fb9e0","Type":"ContainerDied","Data":"1edf6a5e8356cf1d3e7782f73906e9ec7eb70f0f4183c2b476969d438db9d34b"} Oct 29 22:15:20.710682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.707185 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg" Oct 29 22:15:20.720938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.720627 4543 scope.go:122] "RemoveContainer" containerID="765345957b408ab3502bf7a1700e7feb4fd560e02840f237bcee1dd0ef5a2d5b" Oct 29 22:15:20.736153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.736137 4543 scope.go:122] "RemoveContainer" containerID="008b94aa12f65d4222a4cb5e5a1fe51ca071f097af5419c438072ca93450cc37" Oct 29 22:15:20.752991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.752968 4543 scope.go:122] "RemoveContainer" containerID="d1f4e3f151591c9c740a1f2e5c6ae39db1c7910f7309cbe77bd2d5c50e6ec369" Oct 29 22:15:20.767768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.764856 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:20.773067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.773044 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888535-5l64g"] Oct 29 22:15:20.773217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.773198 4543 scope.go:122] "RemoveContainer" containerID="03e621f4c4f3f3b91da7b3b352309fbfbbe676b9ebf096ec3132a4a097c3dd2a" Oct 29 22:15:20.795244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.794330 4543 scope.go:122] "RemoveContainer" containerID="508757bdd00502a15ecb3a7ffa2807b293eb93e856ace6a2b11c7561b71408d0" Oct 29 22:15:20.887128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.887100 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:20.940051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.940010 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:20.941355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.941333 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888535-5zg8s"] Oct 29 22:15:20.981427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.981405 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:20.981856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.981843 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:20.990830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.990803 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888535-29zsq"] Oct 29 22:15:20.992192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.992173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\" (UniqueName: \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\") pod \"647f181c-5a88-44f7-886f-c255f46319c4\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " Oct 29 22:15:20.992379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.992360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\") pod \"647f181c-5a88-44f7-886f-c255f46319c4\" (UID: \"647f181c-5a88-44f7-886f-c255f46319c4\") " Oct 29 22:15:20.992733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.992709 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca" pod "647f181c-5a88-44f7-886f-c255f46319c4" (UID: "647f181c-5a88-44f7-886f-c255f46319c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:20.993264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.993243 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647f181c-5a88-44f7-886f-c255f46319c4-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.994586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.994559 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn" pod "647f181c-5a88-44f7-886f-c255f46319c4" (UID: "647f181c-5a88-44f7-886f-c255f46319c4"). InnerVolumeSpecName "kube-api-access-bhmwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:20.998817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:20.998795 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:21.023137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.023112 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:21.056163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.030040 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888535-8rw9v"] Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.094344 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\" (UniqueName: \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\") pod \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.094466 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\" (UniqueName: \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\") pod \"a71d77a0-9414-40e6-8237-5cb0d7697a41\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.094535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\") pod \"a71d77a0-9414-40e6-8237-5cb0d7697a41\" (UID: \"a71d77a0-9414-40e6-8237-5cb0d7697a41\") " Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.094738 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\") pod \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\" (UID: \"b3d4c4a3-deda-4ad9-a481-d7bf244440cd\") " Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.095821 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmwn\" (UniqueName: \"kubernetes.io/projected/647f181c-5a88-44f7-886f-c255f46319c4-kube-api-access-bhmwn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.097736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.096219 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca" pod "b3d4c4a3-deda-4ad9-a481-d7bf244440cd" (UID: "b3d4c4a3-deda-4ad9-a481-d7bf244440cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:21.100481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.100455 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9" pod "a71d77a0-9414-40e6-8237-5cb0d7697a41" (UID: "a71d77a0-9414-40e6-8237-5cb0d7697a41"). InnerVolumeSpecName "kube-api-access-nc6v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:21.100790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.100767 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca" pod "a71d77a0-9414-40e6-8237-5cb0d7697a41" (UID: "a71d77a0-9414-40e6-8237-5cb0d7697a41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:21.106534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.106501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk" pod "b3d4c4a3-deda-4ad9-a481-d7bf244440cd" (UID: "b3d4c4a3-deda-4ad9-a481-d7bf244440cd"). InnerVolumeSpecName "kube-api-access-khcsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:21.123763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.123721 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:21.128665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.128636 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05687d-29888535-khrqg"] Oct 29 22:15:21.197355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.197280 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khcsk\" (UniqueName: \"kubernetes.io/projected/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-kube-api-access-khcsk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.197355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.197316 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6v9\" (UniqueName: \"kubernetes.io/projected/a71d77a0-9414-40e6-8237-5cb0d7697a41-kube-api-access-nc6v9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.197355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.197334 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d77a0-9414-40e6-8237-5cb0d7697a41-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.197355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.197350 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d4c4a3-deda-4ad9-a481-d7bf244440cd-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.730634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.730606 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" Oct 29 22:15:21.731064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.730602 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp" event={"ID":"b3d4c4a3-deda-4ad9-a481-d7bf244440cd","Type":"ContainerDied","Data":"99cb615be1d21af2ac02939bccd8062b8a2ade7db5afac9f77dc42a15f11203e"} Oct 29 22:15:21.731064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.730763 4543 scope.go:122] "RemoveContainer" containerID="9f3cc92d8841b0111188906716070d483e93b4695dcdacc1096934dfb76b76a3" Oct 29 22:15:21.735904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.735880 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" Oct 29 22:15:21.736023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.735903 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz" event={"ID":"a71d77a0-9414-40e6-8237-5cb0d7697a41","Type":"ContainerDied","Data":"1bcd3c9bc145a09e00dbf48aa9fd8945914df936f63b1a8a07df47e6a93d069c"} Oct 29 22:15:21.737488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.737470 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" Oct 29 22:15:21.737488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.737481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l" event={"ID":"647f181c-5a88-44f7-886f-c255f46319c4","Type":"ContainerDied","Data":"efa13f6719d452e2f1cae2131f65f17a551a8cd036fa85dcbcc52e81e3ee2f15"} Oct 29 22:15:21.745963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.745944 4543 scope.go:122] "RemoveContainer" containerID="1b1b2ace8dab2e823d40c09ce474f894d964d39cceb8996edbc4da7d10b3075a" Oct 29 22:15:21.756097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.756068 4543 scope.go:122] "RemoveContainer" containerID="e275ca685e2e224463d5cd8bd240127e62f659ee6e6e561367de803b62a8f337" Oct 29 22:15:21.826114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.826064 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:21.833479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.833460 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888535-9fsfp"] Oct 29 22:15:21.869185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.869162 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:21.874588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.874567 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888535-b9fhz"] Oct 29 22:15:21.913857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.913834 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:21.919534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:21.919516 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888535-68p4l"] Oct 29 22:15:22.133605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.133544 4543 scope.go:122] "RemoveContainer" containerID="783e751f935d842c76d4f6b73b7dc3fdd23154bd8f33809a8f7ff10f43a3ca71" Oct 29 22:15:22.145673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.145654 4543 scope.go:122] "RemoveContainer" containerID="af0a910a09d0492fa7923a717431f8b415802f736f59ab4d8b19419b97afee52" Oct 29 22:15:22.155878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.155862 4543 scope.go:122] "RemoveContainer" containerID="0f566973eb8fc391de5a594a5281390e0d0b08c2dc6225ae9afafa8545a4ce8c" Oct 29 22:15:22.387911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.387854 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647f181c-5a88-44f7-886f-c255f46319c4" path="/var/lib/kubelet/pods/647f181c-5a88-44f7-886f-c255f46319c4/volumes" Oct 29 22:15:22.388292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.388278 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b12988-8477-44bb-a903-d780202fb9e0" path="/var/lib/kubelet/pods/64b12988-8477-44bb-a903-d780202fb9e0/volumes" Oct 29 22:15:22.388668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.388654 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" path="/var/lib/kubelet/pods/77ce399b-2b7f-4c84-8362-89a4efcfca34/volumes" Oct 29 22:15:22.389351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.389334 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" path="/var/lib/kubelet/pods/a71d77a0-9414-40e6-8237-5cb0d7697a41/volumes" Oct 29 22:15:22.389725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.389712 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" path="/var/lib/kubelet/pods/b3d4c4a3-deda-4ad9-a481-d7bf244440cd/volumes" Oct 29 22:15:22.390208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.390188 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8185d5-4592-4761-b078-7cd63d635020" path="/var/lib/kubelet/pods/da8185d5-4592-4761-b078-7cd63d635020/volumes" Oct 29 22:15:22.390951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.390932 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" path="/var/lib/kubelet/pods/db2f1188-e008-45a9-b06c-6bcb05625eec/volumes" Oct 29 22:15:22.391380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.391364 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2984b7c-87fe-42ce-a046-81066764caad" path="/var/lib/kubelet/pods/e2984b7c-87fe-42ce-a046-81066764caad/volumes" Oct 29 22:15:22.431008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.430982 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:15:22.432341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432315 4543 cpu_manager.go:414] "containerMap: removing container" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432342 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432360 4543 cpu_manager.go:414] "containerMap: removing container" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432367 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432386 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432393 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432407 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432415 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432429 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432436 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432446 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.432456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432453 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432466 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432473 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432494 4543 cpu_manager.go:414] "containerMap: removing container" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432501 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432512 4543 cpu_manager.go:414] "containerMap: removing container" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432519 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432539 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432545 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432555 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432561 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432574 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432581 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432590 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432597 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432610 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432617 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432642 4543 cpu_manager.go:414] "containerMap: removing container" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432649 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432661 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432668 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432678 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432685 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432706 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432713 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432728 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432735 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432748 4543 cpu_manager.go:414] "containerMap: removing container" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432755 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432774 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432780 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432798 4543 cpu_manager.go:414] "containerMap: removing container" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432804 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432814 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432832 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432843 4543 cpu_manager.go:414] "containerMap: removing container" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.432993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.432849 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433017 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433031 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433040 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433052 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433062 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433091 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433100 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433112 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433132 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433143 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433154 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433163 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433171 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433182 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433191 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433200 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8185d5-4592-4761-b078-7cd63d635020" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433207 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433214 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433224 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433236 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433246 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433254 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433266 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433273 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433284 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:22.434821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.433292 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:22.440627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.440607 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.461423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.461329 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:15:22.465459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.465311 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f52917-openshift-default-dockercfg-2q4hk\"" Oct 29 22:15:22.515679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.515650 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.515838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.515690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.515838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.515730 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.515993 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886jh\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.516108 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.516181 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.516217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.516241 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.516390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.516305 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.627274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.627721 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886jh\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.627903 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.627979 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628038 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628094 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628123 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628211 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628256 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.635149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.628984 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.640190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.639511 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.642113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.641981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.646111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.642271 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.646111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.642515 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.649986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.648610 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.706107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.680717 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.751096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.750263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:22.810545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:22.784839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-886jh\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:23.072439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.072133 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:23.562391 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:23.562348 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d2ad4a_9d76_43f9_be12_2d14c6c21e53.slice/crio-6f917318d4e8427386aa0fe49c0c83d9d0b084d9ca7843e583ba6eb57cfda7a8 WatchSource:0}: Error finding container 6f917318d4e8427386aa0fe49c0c83d9d0b084d9ca7843e583ba6eb57cfda7a8: Status 404 returned error can't find the container with id 6f917318d4e8427386aa0fe49c0c83d9d0b084d9ca7843e583ba6eb57cfda7a8 Oct 29 22:15:23.562828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.562804 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:15:23.565333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.565301 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:23.565425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.565362 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:23.565425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.565391 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:23.860153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.859283 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerStarted","Data":"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7"} Oct 29 22:15:23.860153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.859365 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerStarted","Data":"6f917318d4e8427386aa0fe49c0c83d9d0b084d9ca7843e583ba6eb57cfda7a8"} Oct 29 22:15:23.910567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:23.910514 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podStartSLOduration=1.910499954 podStartE2EDuration="1.910499954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:23.908252084 +0000 UTC m=+1985.650182758" watchObservedRunningTime="2026-10-29 22:15:23.910499954 +0000 UTC m=+1985.652430627" Oct 29 22:15:24.077496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:24.075203 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:24.088749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:24.088720 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:24.868218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:24.868179 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:24.869106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:24.869068 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:15:25.040448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:25.040413 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s"] Oct 29 22:15:25.046651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:25.046623 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fb0eaa-p9g5s"] Oct 29 22:15:26.388712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:26.388679 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8be25b-e7fb-4856-8949-6d56c5d70ea3" path="/var/lib/kubelet/pods/6d8be25b-e7fb-4856-8949-6d56c5d70ea3/volumes" Oct 29 22:15:27.041462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:27.041431 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7"] Oct 29 22:15:27.053466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:27.053437 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8674edd5-nj2c7"] Oct 29 22:15:27.962036 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:27.961995 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:15:28.297568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.297499 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:15:28.298315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298297 4543 cpu_manager.go:414] "containerMap: removing container" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298316 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298348 4543 cpu_manager.go:414] "containerMap: removing container" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298353 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04542cbe-3637-4338-a594-49c7406422bf" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298363 4543 cpu_manager.go:414] "containerMap: removing container" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298367 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:28.298379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298380 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298385 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec53711c-2418-4464-bd4a-2ca258fe10e3" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298403 4543 cpu_manager.go:414] "containerMap: removing container" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298407 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298416 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298421 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298428 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298432 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298438 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298442 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298453 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298457 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce399b-2b7f-4c84-8362-89a4efcfca34" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298565 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f1188-e008-45a9-b06c-6bcb05625eec" containerName="cleanup" Oct 29 22:15:28.298572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298573 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71d77a0-9414-40e6-8237-5cb0d7697a41" containerName="cleanup" Oct 29 22:15:28.298954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298582 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d4c4a3-deda-4ad9-a481-d7bf244440cd" containerName="cleanup" Oct 29 22:15:28.298954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298588 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b12988-8477-44bb-a903-d780202fb9e0" containerName="cleanup" Oct 29 22:15:28.298954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298594 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2984b7c-87fe-42ce-a046-81066764caad" containerName="cleanup" Oct 29 22:15:28.298954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298600 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f181c-5a88-44f7-886f-c255f46319c4" containerName="cleanup" Oct 29 22:15:28.298954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.298606 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="71348e81-9a5d-4738-ad0c-561ea7442536" containerName="cleanup" Oct 29 22:15:28.303758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.303742 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.323748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.323724 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:15:28.330676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.330652 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f52917\"" Oct 29 22:15:28.330839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.330733 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f52917-dockercfg-4r66g\"" Oct 29 22:15:28.344045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.343994 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:15:28.348835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.348807 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.507100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.502461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.507100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.502524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.507100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.502607 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2sd\" (UniqueName: \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.507100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.502673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlrk\" (UniqueName: \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.668507 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.668565 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.668644 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2sd\" (UniqueName: \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.668707 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlrk\" (UniqueName: \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.669978 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.682155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.670639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.726346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.721151 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3800f168-65f3-4075-a48a-ac59cbde64a7" path="/var/lib/kubelet/pods/3800f168-65f3-4075-a48a-ac59cbde64a7/volumes" Oct 29 22:15:28.726346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.721699 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:15:28.726346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.723835 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlrk\" (UniqueName: \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\") pod \"payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:28.776860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.766943 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2sd\" (UniqueName: \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\") pod \"payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.920421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.920391 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:28.983932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:28.983885 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:29.315207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.315047 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:15:29.315447 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:29.315414 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5035aff_e5d5_4720_b9ad_5bb32b2cc3ca.slice/crio-d349865dbc8e71ed21e6fff07ef488308acf26d311a614dc8601322007887d34 WatchSource:0}: Error finding container d349865dbc8e71ed21e6fff07ef488308acf26d311a614dc8601322007887d34: Status 404 returned error can't find the container with id d349865dbc8e71ed21e6fff07ef488308acf26d311a614dc8601322007887d34 Oct 29 22:15:29.326134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.326108 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:15:29.330810 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:29.330783 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a0e0fb_64ad_4de5_8b91_c03f74d78f46.slice/crio-193c838df434f11d5b0485da4525cdfeb292becd9ed64389d1866ac8ecb6077b WatchSource:0}: Error finding container 193c838df434f11d5b0485da4525cdfeb292becd9ed64389d1866ac8ecb6077b: Status 404 returned error can't find the container with id 193c838df434f11d5b0485da4525cdfeb292becd9ed64389d1866ac8ecb6077b Oct 29 22:15:29.911848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.911810 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" event={"ID":"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46","Type":"ContainerStarted","Data":"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b"} Oct 29 22:15:29.912057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.911876 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:29.912057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.911900 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" event={"ID":"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46","Type":"ContainerStarted","Data":"193c838df434f11d5b0485da4525cdfeb292becd9ed64389d1866ac8ecb6077b"} Oct 29 22:15:29.913700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.913670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" event={"ID":"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca","Type":"ContainerStarted","Data":"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15"} Oct 29 22:15:29.913826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.913721 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" event={"ID":"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca","Type":"ContainerStarted","Data":"d349865dbc8e71ed21e6fff07ef488308acf26d311a614dc8601322007887d34"} Oct 29 22:15:29.913826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.913750 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:29.961059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:29.961002 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" podStartSLOduration=1.960988054 podStartE2EDuration="1.960988054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:29.954634443 +0000 UTC m=+1991.696565118" watchObservedRunningTime="2026-10-29 22:15:29.960988054 +0000 UTC m=+1991.702918726" Oct 29 22:15:30.000668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:30.000618 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" podStartSLOduration=2.000606554 podStartE2EDuration="2.000606554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:29.997845516 +0000 UTC m=+1991.739776188" watchObservedRunningTime="2026-10-29 22:15:30.000606554 +0000 UTC m=+1991.742537268" Oct 29 22:15:36.037647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:36.037614 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l"] Oct 29 22:15:36.050208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:36.050158 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877eed8c-c9r8l"] Oct 29 22:15:36.390974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:36.390895 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3912a28-cdd8-4299-ab7a-0ca2f63b1075" path="/var/lib/kubelet/pods/a3912a28-cdd8-4299-ab7a-0ca2f63b1075/volumes" Oct 29 22:15:39.585313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.585272 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv"] Oct 29 22:15:39.592684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.592660 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.607064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.607038 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv"] Oct 29 22:15:39.613783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.613744 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:15:39.614292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.614267 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:15:39.614292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.614267 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:15:39.622568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.622549 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:15:39.659330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.659295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.659501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.659483 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.659571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.659519 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bc25\" (UniqueName: \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.760637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.760617 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.760752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.760716 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.760752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.760738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bc25\" (UniqueName: \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.761311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.761291 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.763220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.763200 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.786727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.786703 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bc25\" (UniqueName: \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\") pod \"maas-api-revoke-keys-e2e-autores-971004-5xxrv\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:39.913327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:39.913267 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:40.093917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.093879 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq"] Oct 29 22:15:40.101139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.100280 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804585dd-mlgcq"] Oct 29 22:15:40.421690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.421372 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca4b533-1e91-470e-964e-4ad8e5cc15f5" path="/var/lib/kubelet/pods/8ca4b533-1e91-470e-964e-4ad8e5cc15f5/volumes" Oct 29 22:15:40.473218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.473196 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv"] Oct 29 22:15:40.474609 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:40.474583 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b8b6e9_fb6d_432f_bcbc_8a778641fcad.slice/crio-6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63 WatchSource:0}: Error finding container 6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63: Status 404 returned error can't find the container with id 6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63 Oct 29 22:15:40.925200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.925176 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:15:40.925559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:40.925257 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:15:41.063680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:41.063587 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" event={"ID":"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad","Type":"ContainerStarted","Data":"ceb69a7f06b144a9a60f20c7a72ead1c349cf289cbdcbcac8ad0e49b549c573f"} Oct 29 22:15:41.063680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:41.063655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" event={"ID":"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad","Type":"ContainerStarted","Data":"6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63"} Oct 29 22:15:41.115052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:41.115007 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" podStartSLOduration=2.114993708 podStartE2EDuration="2.114993708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:41.112625887 +0000 UTC m=+2002.854556562" watchObservedRunningTime="2026-10-29 22:15:41.114993708 +0000 UTC m=+2002.856924380" Oct 29 22:15:42.075704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.075672 4543 generic.go:356] "Generic (PLEG): container finished" podID="e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" containerID="ceb69a7f06b144a9a60f20c7a72ead1c349cf289cbdcbcac8ad0e49b549c573f" exitCode=0 Oct 29 22:15:42.076124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.075774 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" event={"ID":"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad","Type":"ContainerDied","Data":"ceb69a7f06b144a9a60f20c7a72ead1c349cf289cbdcbcac8ad0e49b549c573f"} Oct 29 22:15:42.907910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.907877 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:15:42.913777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.913752 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.934937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935003 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935060 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935201 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935301 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935339 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935368 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzfr\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935417 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.938128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.935499 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:42.939416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.939390 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:15:42.973695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:42.973670 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-798a8f-openshift-default-dockercfg-v4xv7\"" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079538 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079683 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079825 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079940 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.079979 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.080008 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzfr\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.080092 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.080194 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.080798 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.081930 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.082589 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.083102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.082831 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.134098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.131231 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.134098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.083061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.187250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.178615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.187250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.179806 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzfr\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.187250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.180058 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\") pod \"e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.254113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.249395 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:43.467176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.467148 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:43.615717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.615684 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\") pod \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " Oct 29 22:15:43.615890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.615860 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\") pod \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " Oct 29 22:15:43.615954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.615921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\" (UniqueName: \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\") pod \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\" (UID: \"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad\") " Oct 29 22:15:43.616337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.616257 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca" pod "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" (UID: "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:43.616592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.616506 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:43.617890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.617867 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca" pod "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" (UID: "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:43.618155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.618131 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25" pod "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" (UID: "e6b8b6e9-fb6d-432f-bcbc-8a778641fcad"). InnerVolumeSpecName "kube-api-access-7bc25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:43.631377 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:43.631354 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc513c5dc_9256_40cf_a616_ada014206f81.slice/crio-19f50e2fbfd4342cb25ba1161ecb3c2c006699a1e1b0779f6029178d3519e6c5 WatchSource:0}: Error finding container 19f50e2fbfd4342cb25ba1161ecb3c2c006699a1e1b0779f6029178d3519e6c5: Status 404 returned error can't find the container with id 19f50e2fbfd4342cb25ba1161ecb3c2c006699a1e1b0779f6029178d3519e6c5 Oct 29 22:15:43.632445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.632423 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:15:43.634648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.634617 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:43.634741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.634687 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:43.634741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.634722 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:15:43.717759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.717737 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bc25\" (UniqueName: \"kubernetes.io/projected/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-kube-api-access-7bc25\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:43.717856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:43.717762 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:44.230091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.230044 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" Oct 29 22:15:44.230471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.230105 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv" event={"ID":"e6b8b6e9-fb6d-432f-bcbc-8a778641fcad","Type":"ContainerDied","Data":"6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63"} Oct 29 22:15:44.230471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.230151 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4ff3fc851293e97b16c81d3d1ba067143975748c5b0a2ed4502401466a8c63" Oct 29 22:15:44.234086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.234043 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerStarted","Data":"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc"} Oct 29 22:15:44.234218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.234132 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerStarted","Data":"19f50e2fbfd4342cb25ba1161ecb3c2c006699a1e1b0779f6029178d3519e6c5"} Oct 29 22:15:44.250230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.250208 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:44.255885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.255866 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:44.285759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:44.285716 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podStartSLOduration=2.285702699 podStartE2EDuration="2.285702699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:44.284516146 +0000 UTC m=+2006.026446817" watchObservedRunningTime="2026-10-29 22:15:44.285702699 +0000 UTC m=+2006.027633372" Oct 29 22:15:45.244085 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:45.244054 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:45.245012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:45.244994 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:15:48.703219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.703187 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:15:48.704012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.703997 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" containerName="revoke-keys" Oct 29 22:15:48.704058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.704013 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" containerName="revoke-keys" Oct 29 22:15:48.704164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.704149 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" containerName="revoke-keys" Oct 29 22:15:48.708312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.708296 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:48.726056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.726032 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:15:48.733299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.733274 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-798a8f\"" Oct 29 22:15:48.733424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.733364 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-798a8f-dockercfg-hrpsx\"" Oct 29 22:15:48.842099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.824508 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8v27\" (UniqueName: \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:48.842099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.824681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:48.937154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.930980 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8v27\" (UniqueName: \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:48.937154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.931170 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:48.937154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:48.932404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:49.075990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.075949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8v27\" (UniqueName: \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\") pod \"payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:49.081466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.081441 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:15:49.115505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.115469 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:15:49.115698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.115609 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.151105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.151055 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.151305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.151286 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjvz\" (UniqueName: \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.253188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.253160 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.253374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.253357 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjvz\" (UniqueName: \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.253799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.253779 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.280569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.280544 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjvz\" (UniqueName: \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\") pod \"payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.323592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.323572 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:49.434719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.434691 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:49.644230 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:49.644203 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e769a2_04c7_4b06_8cff_5ec63fbd2b4b.slice/crio-51e5acd41cd9de99d90813a97d7e8d0b6e792c1f374f56809830a5d19e9c3987 WatchSource:0}: Error finding container 51e5acd41cd9de99d90813a97d7e8d0b6e792c1f374f56809830a5d19e9c3987: Status 404 returned error can't find the container with id 51e5acd41cd9de99d90813a97d7e8d0b6e792c1f374f56809830a5d19e9c3987 Oct 29 22:15:49.645593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.645570 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:15:49.765393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:49.765352 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:15:49.765393 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:15:49.765369 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7e2b4d_6057_4430_9879_69bc6060fc9e.slice/crio-aae872f377386830093c1cbd619594e231ba2e55492a46b3c7a23259052ab963 WatchSource:0}: Error finding container aae872f377386830093c1cbd619594e231ba2e55492a46b3c7a23259052ab963: Status 404 returned error can't find the container with id aae872f377386830093c1cbd619594e231ba2e55492a46b3c7a23259052ab963 Oct 29 22:15:50.095659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.095628 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:15:50.142163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.142135 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:15:50.317871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.317825 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" event={"ID":"4b7e2b4d-6057-4430-9879-69bc6060fc9e","Type":"ContainerStarted","Data":"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1"} Oct 29 22:15:50.318057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.317894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" event={"ID":"4b7e2b4d-6057-4430-9879-69bc6060fc9e","Type":"ContainerStarted","Data":"aae872f377386830093c1cbd619594e231ba2e55492a46b3c7a23259052ab963"} Oct 29 22:15:50.318057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.317926 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:15:50.319748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.319707 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" event={"ID":"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b","Type":"ContainerStarted","Data":"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192"} Oct 29 22:15:50.319876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.319749 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" event={"ID":"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b","Type":"ContainerStarted","Data":"51e5acd41cd9de99d90813a97d7e8d0b6e792c1f374f56809830a5d19e9c3987"} Oct 29 22:15:50.319876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.319841 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:15:50.406228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.382496 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" podStartSLOduration=2.382474032 podStartE2EDuration="2.382474032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:50.363790874 +0000 UTC m=+2012.105721547" watchObservedRunningTime="2026-10-29 22:15:50.382474032 +0000 UTC m=+2012.124404707" Oct 29 22:15:50.423916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:50.423846 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" podStartSLOduration=2.42382786 podStartE2EDuration="2.42382786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:50.419839825 +0000 UTC m=+2012.161770576" watchObservedRunningTime="2026-10-29 22:15:50.42382786 +0000 UTC m=+2012.165758532" Oct 29 22:15:55.096372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.096317 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" podUID="87774e3d-4618-4b45-9512-5296df11c1cd" containerName="payload-processing" containerID="cri-o://fc7d35cd87168dca6471938c4ce0477d52049d20265dfbd7c99f2a0f1eddad1a" gracePeriod=25 Oct 29 22:15:55.143107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.143056 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" containerName="payload-pre-processing" containerID="cri-o://4c5c955f3cf9a255e49d467ba50ae3c09acce2967a7cb59ad33d1df4d688fd3e" gracePeriod=25 Oct 29 22:15:55.371646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.371610 4543 generic.go:356] "Generic (PLEG): container finished" podID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" containerID="4c5c955f3cf9a255e49d467ba50ae3c09acce2967a7cb59ad33d1df4d688fd3e" exitCode=0 Oct 29 22:15:55.371850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.371832 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" event={"ID":"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9","Type":"ContainerDied","Data":"4c5c955f3cf9a255e49d467ba50ae3c09acce2967a7cb59ad33d1df4d688fd3e"} Oct 29 22:15:55.373968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.373942 4543 generic.go:356] "Generic (PLEG): container finished" podID="87774e3d-4618-4b45-9512-5296df11c1cd" containerID="fc7d35cd87168dca6471938c4ce0477d52049d20265dfbd7c99f2a0f1eddad1a" exitCode=0 Oct 29 22:15:55.374106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.374011 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" event={"ID":"87774e3d-4618-4b45-9512-5296df11c1cd","Type":"ContainerDied","Data":"fc7d35cd87168dca6471938c4ce0477d52049d20265dfbd7c99f2a0f1eddad1a"} Oct 29 22:15:55.461351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.461322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:15:55.489512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.489492 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:15:55.605788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.605759 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\" (UniqueName: \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\") pod \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " Oct 29 22:15:55.606058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.606041 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\") pod \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\" (UID: \"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9\") " Oct 29 22:15:55.607163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.606232 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\") pod \"87774e3d-4618-4b45-9512-5296df11c1cd\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " Oct 29 22:15:55.607163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.606281 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\" (UniqueName: \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\") pod \"87774e3d-4618-4b45-9512-5296df11c1cd\" (UID: \"87774e3d-4618-4b45-9512-5296df11c1cd\") " Oct 29 22:15:55.607163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.606501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume" pod "719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" (UID: "719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:55.607163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.606870 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume" pod "87774e3d-4618-4b45-9512-5296df11c1cd" (UID: "87774e3d-4618-4b45-9512-5296df11c1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:55.607163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.607151 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:55.607443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.607173 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87774e3d-4618-4b45-9512-5296df11c1cd-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:55.608668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.608642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm" pod "719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" (UID: "719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9"). InnerVolumeSpecName "kube-api-access-bpxzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:55.609177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.609156 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69" pod "87774e3d-4618-4b45-9512-5296df11c1cd" (UID: "87774e3d-4618-4b45-9512-5296df11c1cd"). InnerVolumeSpecName "kube-api-access-bxl69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:55.716153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.711998 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxl69\" (UniqueName: \"kubernetes.io/projected/87774e3d-4618-4b45-9512-5296df11c1cd-kube-api-access-bxl69\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:55.716153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:55.712037 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxzm\" (UniqueName: \"kubernetes.io/projected/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9-kube-api-access-bpxzm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:15:56.385283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.385210 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" Oct 29 22:15:56.387138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.387117 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:15:56.389259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.389235 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q" event={"ID":"87774e3d-4618-4b45-9512-5296df11c1cd","Type":"ContainerDied","Data":"49c29ef767d18d23f3444971aeb2bdf118e29022190c331e77e0546396c00f28"} Oct 29 22:15:56.389367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.389279 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" event={"ID":"719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9","Type":"ContainerDied","Data":"718157c460e041a27a72f16bcb6960ffa8b50c904e6d057e356059f6a59d313d"} Oct 29 22:15:56.389367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.389304 4543 scope.go:122] "RemoveContainer" containerID="fc7d35cd87168dca6471938c4ce0477d52049d20265dfbd7c99f2a0f1eddad1a" Oct 29 22:15:56.404572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.404550 4543 scope.go:122] "RemoveContainer" containerID="4c5c955f3cf9a255e49d467ba50ae3c09acce2967a7cb59ad33d1df4d688fd3e" Oct 29 22:15:56.445102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.445066 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:15:56.457761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:56.457730 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-971004-c7c7c95c4-lqf6q"] Oct 29 22:15:57.962717 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:15:57.962682 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:15:58.038031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:58.038002 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld"] Oct 29 22:15:58.050837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:58.050813 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d3b57c-9gnld"] Oct 29 22:15:58.388226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:58.388157 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87774e3d-4618-4b45-9512-5296df11c1cd" path="/var/lib/kubelet/pods/87774e3d-4618-4b45-9512-5296df11c1cd/volumes" Oct 29 22:15:58.388640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:58.388626 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb" path="/var/lib/kubelet/pods/fc90f2df-f022-4ad1-b5d6-d1dca22c7ebb/volumes" Oct 29 22:15:59.038842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:59.038785 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v"] Oct 29 22:15:59.056123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:15:59.054165 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7e49bc-6vc4v"] Oct 29 22:16:00.069274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:00.069240 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv"] Oct 29 22:16:00.082209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:00.082177 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-971004-5xxrv"] Oct 29 22:16:00.403515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:00.403428 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94014a6d-f044-4a37-ad88-61ff872ec68c" path="/var/lib/kubelet/pods/94014a6d-f044-4a37-ad88-61ff872ec68c/volumes" Oct 29 22:16:00.404246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:00.404224 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b8b6e9-fb6d-432f-bcbc-8a778641fcad" path="/var/lib/kubelet/pods/e6b8b6e9-fb6d-432f-bcbc-8a778641fcad/volumes" Oct 29 22:16:01.332125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:01.332093 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:16:01.332490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:01.332168 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:16:08.538506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:08.538470 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:16:08.539038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:08.538684 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" containerID="cri-o://1e9ca86ba0c58ef10f2012b0c9c9ff1764809dcc4986dd105667754ca61f3446" gracePeriod=30 Oct 29 22:16:08.780103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:08.775267 4543 patch_prober.go:29] interesting pod/e2e-autores-971004-openshift-default-64cd89f865-rp4mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:16:08.780103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:08.775346 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:16:11.051419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:11.051331 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x"] Oct 29 22:16:11.064282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:11.064255 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d62f35-l4r2x"] Oct 29 22:16:12.388011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:12.387970 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c64b6c-87e5-4654-a583-9fe76e42475c" path="/var/lib/kubelet/pods/01c64b6c-87e5-4654-a583-9fe76e42475c/volumes" Oct 29 22:16:13.735923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:13.735890 4543 generic.go:356] "Generic (PLEG): container finished" podID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerID="1e9ca86ba0c58ef10f2012b0c9c9ff1764809dcc4986dd105667754ca61f3446" exitCode=0 Oct 29 22:16:13.736427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:13.736011 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerDied","Data":"1e9ca86ba0c58ef10f2012b0c9c9ff1764809dcc4986dd105667754ca61f3446"} Oct 29 22:16:13.737138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:13.736879 4543 scope.go:122] "RemoveContainer" containerID="7c9e54eaf236ac6a6e3f297d83f26dde56fe9be33dc15b505a3642ea27ecaddd" Oct 29 22:16:13.891612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:13.891587 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:16:14.018919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.018887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019039 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019119 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019191 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019249 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019376 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019383 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:14.019536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019411 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019461 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019530 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\") pod \"cbc56a89-6948-401e-bf3f-b1be24f7c591\" (UID: \"cbc56a89-6948-401e-bf3f-b1be24f7c591\") " Oct 29 22:16:14.019704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.019624 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:14.020270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020156 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:14.020270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020181 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:14.020270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020247 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:14.020584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020447 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.020584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020467 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.020584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020497 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc56a89-6948-401e-bf3f-b1be24f7c591-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.020584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020512 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.020584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.020562 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.022329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.022304 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:14.022879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.022853 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:16:14.023013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.022952 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:14.023533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.023510 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2" pod "cbc56a89-6948-401e-bf3f-b1be24f7c591" (UID: "cbc56a89-6948-401e-bf3f-b1be24f7c591"). InnerVolumeSpecName "kube-api-access-wz9q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:14.122740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.122655 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.122740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.122686 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.122740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.122699 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc56a89-6948-401e-bf3f-b1be24f7c591-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.122740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.122711 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9q2\" (UniqueName: \"kubernetes.io/projected/cbc56a89-6948-401e-bf3f-b1be24f7c591-kube-api-access-wz9q2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:14.747146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.747109 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" event={"ID":"cbc56a89-6948-401e-bf3f-b1be24f7c591","Type":"ContainerDied","Data":"d5bcfff09a9ee4e8d737d2c8192a14918efd4a6144c193c03a41b17e28f2afbe"} Oct 29 22:16:14.747146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.747140 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj" Oct 29 22:16:14.747658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.747167 4543 scope.go:122] "RemoveContainer" containerID="1e9ca86ba0c58ef10f2012b0c9c9ff1764809dcc4986dd105667754ca61f3446" Oct 29 22:16:14.824929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.824896 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:16:14.837494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:14.837461 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-971004-openshift-default-64cd89f865-rp4mj"] Oct 29 22:16:16.469778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:16.466680 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" path="/var/lib/kubelet/pods/cbc56a89-6948-401e-bf3f-b1be24f7c591/volumes" Oct 29 22:16:19.073377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:19.072393 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8"] Oct 29 22:16:19.083190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:19.079301 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53371b15-skts8"] Oct 29 22:16:20.388212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:20.388179 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facd82c7-d7d9-4e97-88bc-165887719d45" path="/var/lib/kubelet/pods/facd82c7-d7d9-4e97-88bc-165887719d45/volumes" Oct 29 22:16:22.263945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.263911 4543 scope.go:122] "RemoveContainer" containerID="4b5487a28253ebcdf76ee636cfaf8a5df35dacd0d30bacc776c09b3bb706af5d" Oct 29 22:16:22.274669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.274645 4543 scope.go:122] "RemoveContainer" containerID="2fa7fda13d268f9f773ccfdc86f1ce767d0e82c5635d2fd4ef92b90be7fb0e66" Oct 29 22:16:22.284470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.284437 4543 scope.go:122] "RemoveContainer" containerID="8da5d1cdf55ce49e34e65a3af8bceeead58a7c561408200bf6ec874281247cbf" Oct 29 22:16:22.294323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.294298 4543 scope.go:122] "RemoveContainer" containerID="b78f969dccd54dbb23efebd5a53c5bf11c0e007259b6614693adf21c88191360" Oct 29 22:16:22.303719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.303703 4543 scope.go:122] "RemoveContainer" containerID="12d110cb8bdbbde2c7b67cf8c971a4a3d7344b57da594b024cfaf32fc756564b" Oct 29 22:16:22.312905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.312887 4543 scope.go:122] "RemoveContainer" containerID="cee23880f3fdcdae9c4b9d9e0f9c3524d71a5ab18521bdba1c579b788eee75fb" Oct 29 22:16:22.322373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.322356 4543 scope.go:122] "RemoveContainer" containerID="8acd857698394938a1cbf89f39c8431db00dc2f60ce6d53909983fb891768a74" Oct 29 22:16:22.332700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.332673 4543 scope.go:122] "RemoveContainer" containerID="25babdf5505e6325ee657034bcaeff13e6f032e20e7b140e6b648014f689aecc" Oct 29 22:16:22.345468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.345443 4543 scope.go:122] "RemoveContainer" containerID="23548bf032dca6308f3a56d74af10a79a4331cd2ddfe19d7bdd1f8b3753e696d" Oct 29 22:16:22.356801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:22.356776 4543 scope.go:122] "RemoveContainer" containerID="b173402577f461b7c659636ab54b6ecf7ebe910ad57e8ea4acaae7ce5b84cb4e" Oct 29 22:16:26.399889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:26.399846 4543 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9] : Timed out while waiting for systemd to remove kubepods-burstable-pod719ea2db_6ed7_4bd0_88ba_d2bc5e3be0c9.slice" Oct 29 22:16:26.399889 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:16:26.399883 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9] : unable to destroy cgroup paths for cgroup [kubepods burstable pod719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9] : Timed out while waiting for systemd to remove kubepods-burstable-pod719ea2db_6ed7_4bd0_88ba_d2bc5e3be0c9.slice" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" Oct 29 22:16:26.867867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:26.867833 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp" Oct 29 22:16:26.953042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:26.953013 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:16:26.958379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:26.958356 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-971004-6dc6f6f495-9jqpp"] Oct 29 22:16:27.961804 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:16:27.961767 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:16:28.398219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:28.398143 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" path="/var/lib/kubelet/pods/719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9/volumes" Oct 29 22:16:34.087643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:34.087613 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26"] Oct 29 22:16:34.101147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:34.101121 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977774f-n9l26"] Oct 29 22:16:34.387956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:34.387883 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95880d21-2ff4-44e8-a166-a23e769bea44" path="/var/lib/kubelet/pods/95880d21-2ff4-44e8-a166-a23e769bea44/volumes" Oct 29 22:16:36.348354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:36.348324 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:16:36.348728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:36.348540 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" containerID="cri-o://1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a" gracePeriod=30 Oct 29 22:16:36.363170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:36.363143 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:16:36.363360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:36.363340 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" containerID="cri-o://ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72" gracePeriod=30 Oct 29 22:16:40.035953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:40.035922 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29"] Oct 29 22:16:40.048855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:40.048826 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d58095-dfd29"] Oct 29 22:16:40.387621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:40.387541 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655" path="/var/lib/kubelet/pods/c3dc3dbd-c2f3-40d8-8a86-a98d81dc0655/volumes" Oct 29 22:16:41.657284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.657250 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:16:41.719418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.719391 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:16:41.801790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.801761 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.802108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802063 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.802433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802397 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.802668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802649 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.802832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802793 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:41.802993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802909 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.802993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.802958 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.803159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803046 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.803159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803116 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.803299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803204 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.803299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803232 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\") pod \"2f7a6a14-5efa-4717-b459-fa772fb664aa\" (UID: \"2f7a6a14-5efa-4717-b459-fa772fb664aa\") " Oct 29 22:16:41.803413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803386 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.803564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803545 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.803981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803965 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.804045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.803988 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7a6a14-5efa-4717-b459-fa772fb664aa-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.804045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.804005 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.804045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.804018 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.817676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.807170 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.817676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.809344 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:41.817676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.814042 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:16:41.817676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.815896 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "kube-api-access-rj7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:41.820955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.820930 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy" pod "2f7a6a14-5efa-4717-b459-fa772fb664aa" (UID: "2f7a6a14-5efa-4717-b459-fa772fb664aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.905050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.904971 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905037 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905093 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905152 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905200 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905243 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905327 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905390 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.905542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905459 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:41.905542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905467 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.905542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905480 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.905793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.905679 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\") pod \"811345cd-c9e3-4159-a083-6f2749461f3c\" (UID: \"811345cd-c9e3-4159-a083-6f2749461f3c\") " Oct 29 22:16:41.906145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.906016 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.906290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.906163 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.906970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.906944 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7r9\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-kube-api-access-rj7r9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.906976 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.906993 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907008 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811345cd-c9e3-4159-a083-6f2749461f3c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907021 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907034 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907046 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907060 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907093 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7a6a14-5efa-4717-b459-fa772fb664aa-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907108 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7a6a14-5efa-4717-b459-fa772fb664aa-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:41.907701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907603 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:16:41.907765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907700 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "kube-api-access-bxh5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:41.907765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:41.907927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:41.907762 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token" pod "811345cd-c9e3-4159-a083-6f2749461f3c" (UID: "811345cd-c9e3-4159-a083-6f2749461f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:42.008936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.008908 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh5t\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-kube-api-access-bxh5t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:42.008936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.008931 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811345cd-c9e3-4159-a083-6f2749461f3c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:42.008936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.008943 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811345cd-c9e3-4159-a083-6f2749461f3c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:42.009239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.008954 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811345cd-c9e3-4159-a083-6f2749461f3c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:16:42.015924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.015901 4543 generic.go:356] "Generic (PLEG): container finished" podID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerID="1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a" exitCode=0 Oct 29 22:16:42.016024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.015948 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" Oct 29 22:16:42.016024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.015984 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" event={"ID":"2f7a6a14-5efa-4717-b459-fa772fb664aa","Type":"ContainerDied","Data":"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a"} Oct 29 22:16:42.016124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.016045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6" event={"ID":"2f7a6a14-5efa-4717-b459-fa772fb664aa","Type":"ContainerDied","Data":"49ff22db7209a7ebb1e9a8a610d922d2ece1a8049601a83147fabe3df2861473"} Oct 29 22:16:42.016124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.016067 4543 scope.go:122] "RemoveContainer" containerID="1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a" Oct 29 22:16:42.022300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.022276 4543 generic.go:356] "Generic (PLEG): container finished" podID="811345cd-c9e3-4159-a083-6f2749461f3c" containerID="ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72" exitCode=0 Oct 29 22:16:42.022415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.022323 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" Oct 29 22:16:42.022415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.022346 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" event={"ID":"811345cd-c9e3-4159-a083-6f2749461f3c","Type":"ContainerDied","Data":"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72"} Oct 29 22:16:42.022415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.022399 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg" event={"ID":"811345cd-c9e3-4159-a083-6f2749461f3c","Type":"ContainerDied","Data":"5edcf4b0b6087c07bb24e8302de74304a367168c5d09295e78e82a19a9f72f7d"} Oct 29 22:16:42.033238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.033220 4543 scope.go:122] "RemoveContainer" containerID="1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a" Oct 29 22:16:42.033556 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:16:42.033536 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a\": container with ID starting with 1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a not found: ID does not exist" containerID="1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a" Oct 29 22:16:42.033599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.033564 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a"} err="failed to get container status \"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a\": rpc error: code = NotFound desc = could not find container \"1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a\": container with ID starting with 1598eee251acdc9163a10ca6905180c2a4fa11a7d4994345fe159a88e9f6925a not found: ID does not exist" Oct 29 22:16:42.033599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.033581 4543 scope.go:122] "RemoveContainer" containerID="ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72" Oct 29 22:16:42.043860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.043838 4543 scope.go:122] "RemoveContainer" containerID="ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72" Oct 29 22:16:42.044166 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:16:42.044147 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72\": container with ID starting with ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72 not found: ID does not exist" containerID="ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72" Oct 29 22:16:42.044240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.044173 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72"} err="failed to get container status \"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72\": rpc error: code = NotFound desc = could not find container \"ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72\": container with ID starting with ac379b04029833c704e7219d79decfcc161dbbf3cea3807976a74c1a08810d72 not found: ID does not exist" Oct 29 22:16:42.076845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.076825 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:16:42.084354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.084336 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-w8ffg"] Oct 29 22:16:42.139854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.139815 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:16:42.145628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.145597 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-945v6"] Oct 29 22:16:42.388594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.388563 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" path="/var/lib/kubelet/pods/2f7a6a14-5efa-4717-b459-fa772fb664aa/volumes" Oct 29 22:16:42.389045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:42.389030 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" path="/var/lib/kubelet/pods/811345cd-c9e3-4159-a083-6f2749461f3c/volumes" Oct 29 22:16:45.726780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.726746 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:16:45.728037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728013 4543 cpu_manager.go:414] "containerMap: removing container" podUID="87774e3d-4618-4b45-9512-5296df11c1cd" containerName="payload-processing" Oct 29 22:16:45.728037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728038 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87774e3d-4618-4b45-9512-5296df11c1cd" containerName="payload-processing" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728057 4543 cpu_manager.go:414] "containerMap: removing container" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" containerName="payload-pre-processing" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728065 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" containerName="payload-pre-processing" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728123 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728132 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728180 4543 cpu_manager.go:414] "containerMap: removing container" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728188 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728198 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" Oct 29 22:16:45.728231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728206 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" Oct 29 22:16:45.728489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728392 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:45.728489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728405 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="811345cd-c9e3-4159-a083-6f2749461f3c" containerName="istio-proxy" Oct 29 22:16:45.728489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728417 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7a6a14-5efa-4717-b459-fa772fb664aa" containerName="istio-proxy" Oct 29 22:16:45.728489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728431 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="87774e3d-4618-4b45-9512-5296df11c1cd" containerName="payload-processing" Oct 29 22:16:45.728489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.728442 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="719ea2db-6ed7-4bd0-88ba-d2bc5e3be0c9" containerName="payload-pre-processing" Oct 29 22:16:45.734761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.734719 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.759685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.759658 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.759925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.759903 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.760865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.760843 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761382 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761489 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761599 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761634 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4jn\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761865 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.761935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.761916 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.768098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.768045 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:16:45.773059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.773029 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b7c2d5-openshift-default-dockercfg-tnqlg\"" Oct 29 22:16:45.989160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.965530 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.989498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.989461 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:45.994101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:45.991330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.014908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.014879 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.015756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.015567 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.022570 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026197 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026232 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026288 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026388 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026490 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.027162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.026526 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4jn\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.041114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.027718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.041114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.028368 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.041114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.028604 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.041114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.028837 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.041114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.033015 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.100302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.086469 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4jn\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\") pod \"e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.100302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.086766 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:46.461954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.461925 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:16:46.462464 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:16:46.462439 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9560df2f_92d0_474d_b279_283cb493c35c.slice/crio-0dd9aa2b382afcc0adc3ec0f4ea32813a91a967afa21049eb87618b9b98efc50 WatchSource:0}: Error finding container 0dd9aa2b382afcc0adc3ec0f4ea32813a91a967afa21049eb87618b9b98efc50: Status 404 returned error can't find the container with id 0dd9aa2b382afcc0adc3ec0f4ea32813a91a967afa21049eb87618b9b98efc50 Oct 29 22:16:46.465103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.465052 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:16:46.465167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.465141 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:16:46.465210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:46.465173 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:16:47.133098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:47.133045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerStarted","Data":"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0"} Oct 29 22:16:47.133566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:47.133129 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerStarted","Data":"0dd9aa2b382afcc0adc3ec0f4ea32813a91a967afa21049eb87618b9b98efc50"} Oct 29 22:16:47.209796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:47.209740 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podStartSLOduration=2.209723122 podStartE2EDuration="2.209723122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:47.205592882 +0000 UTC m=+2068.947523554" watchObservedRunningTime="2026-10-29 22:16:47.209723122 +0000 UTC m=+2068.951653796" Oct 29 22:16:48.088688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:48.088653 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:48.094908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:48.094883 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:48.142070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:48.142045 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:48.142916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:48.142898 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:16:50.814607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.814572 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:16:50.816294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.816264 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:50.816294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.816290 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:50.816575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.816527 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc56a89-6948-401e-bf3f-b1be24f7c591" containerName="istio-proxy" Oct 29 22:16:50.821262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.821209 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.851227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.848963 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-8d569e\"/\"auth-test-model-8d569e-kserve-self-signed-certs\"" Oct 29 22:16:50.851227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.849125 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-8d569e\"/\"default-dockercfg-qcwn5\"" Oct 29 22:16:50.854669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.854644 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-8d569e\"/\"openshift-service-ca.crt\"" Oct 29 22:16:50.854936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.854918 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-8d569e\"/\"kube-root-ca.crt\"" Oct 29 22:16:50.868492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.868462 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:16:50.925613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96vd\" (UniqueName: \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.925728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.925778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925740 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.925824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925792 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.925928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925907 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:50.925973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:50.925947 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96vd\" (UniqueName: \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027467 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027541 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027597 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027650 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.027719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.027689 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.028030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.028004 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.028183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.028040 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.028183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.028123 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.029997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.029959 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.030303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.030286 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.053007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.052981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96vd\" (UniqueName: \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\") pod \"auth-test-model-8d569e-kserve-584f965ccc-nk2jc\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.186846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.182027 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:51.529314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:51.529285 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:16:51.532965 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:16:51.532931 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca1055db_7b3e_475a_8c38_5a024c82a40b.slice/crio-996025980c367e953080695a6633e00e7fe716abaae17b475e5ed9169fa2083a WatchSource:0}: Error finding container 996025980c367e953080695a6633e00e7fe716abaae17b475e5ed9169fa2083a: Status 404 returned error can't find the container with id 996025980c367e953080695a6633e00e7fe716abaae17b475e5ed9169fa2083a Oct 29 22:16:52.192454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.192418 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" event={"ID":"ca1055db-7b3e-475a-8c38-5a024c82a40b","Type":"ContainerStarted","Data":"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c"} Oct 29 22:16:52.192880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.192470 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" event={"ID":"ca1055db-7b3e-475a-8c38-5a024c82a40b","Type":"ContainerStarted","Data":"996025980c367e953080695a6633e00e7fe716abaae17b475e5ed9169fa2083a"} Oct 29 22:16:52.243864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.243817 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podStartSLOduration=2.243804326 podStartE2EDuration="2.243804326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:52.240214711 +0000 UTC m=+2073.982145384" watchObservedRunningTime="2026-10-29 22:16:52.243804326 +0000 UTC m=+2073.985735015" Oct 29 22:16:52.538952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.538917 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:16:52.544757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.544737 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:52.562704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.562684 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b7c2d5\"" Oct 29 22:16:52.562920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.562906 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b7c2d5-dockercfg-ts6gz\"" Oct 29 22:16:52.563016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.562996 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:16:52.627622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.627488 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:16:52.633302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.633261 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.751101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.714166 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.755308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.752010 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xxb\" (UniqueName: \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:52.757957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.756309 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:52.806149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.802369 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqj7\" (UniqueName: \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.954830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.954779 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqj7\" (UniqueName: \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.955370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.955342 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.955587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.955574 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xxb\" (UniqueName: \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:52.955847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.955807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:52.977233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.975845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:52.979602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:52.979402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:53.033236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.031033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqj7\" (UniqueName: \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\") pod \"payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:53.094669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.094292 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:53.101881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.101840 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:16:53.135313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.129933 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xxb\" (UniqueName: \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\") pod \"payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:53.188834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.185026 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:53.188834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.187808 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:16:53.538353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.538322 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:16:53.538709 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:16:53.538666 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb737f7de_d607_4f55_9f84_4cf609efd309.slice/crio-12224b6b9de43acd243fe8d0fd1bde0a8289b2591a2fba15a5a6b87150f5fe9c WatchSource:0}: Error finding container 12224b6b9de43acd243fe8d0fd1bde0a8289b2591a2fba15a5a6b87150f5fe9c: Status 404 returned error can't find the container with id 12224b6b9de43acd243fe8d0fd1bde0a8289b2591a2fba15a5a6b87150f5fe9c Oct 29 22:16:53.581355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:53.581331 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:16:53.583237 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:16:53.583209 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a95c554_1eb4_4b51_a024_79d4e1667e9c.slice/crio-67722ba855de31d55cf3b70c56dd7f13098980a6386dd96fc68042a266bb60da WatchSource:0}: Error finding container 67722ba855de31d55cf3b70c56dd7f13098980a6386dd96fc68042a266bb60da: Status 404 returned error can't find the container with id 67722ba855de31d55cf3b70c56dd7f13098980a6386dd96fc68042a266bb60da Oct 29 22:16:54.230933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.230894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" event={"ID":"b737f7de-d607-4f55-9f84-4cf609efd309","Type":"ContainerStarted","Data":"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb"} Oct 29 22:16:54.231201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.230954 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" event={"ID":"b737f7de-d607-4f55-9f84-4cf609efd309","Type":"ContainerStarted","Data":"12224b6b9de43acd243fe8d0fd1bde0a8289b2591a2fba15a5a6b87150f5fe9c"} Oct 29 22:16:54.231201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.230976 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:16:54.232807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.232782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" event={"ID":"8a95c554-1eb4-4b51-a024-79d4e1667e9c","Type":"ContainerStarted","Data":"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490"} Oct 29 22:16:54.232937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.232839 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" event={"ID":"8a95c554-1eb4-4b51-a024-79d4e1667e9c","Type":"ContainerStarted","Data":"67722ba855de31d55cf3b70c56dd7f13098980a6386dd96fc68042a266bb60da"} Oct 29 22:16:54.232937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.232875 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:16:54.276956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.276911 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" podStartSLOduration=2.27689916 podStartE2EDuration="2.27689916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:54.27478909 +0000 UTC m=+2076.016719789" watchObservedRunningTime="2026-10-29 22:16:54.27689916 +0000 UTC m=+2076.018829832" Oct 29 22:16:54.322804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:16:54.322758 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" podStartSLOduration=2.322746909 podStartE2EDuration="2.322746909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:54.318770782 +0000 UTC m=+2076.060701454" watchObservedRunningTime="2026-10-29 22:16:54.322746909 +0000 UTC m=+2076.064677582" Oct 29 22:16:58.027900 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:16:58.027863 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:17:01.189103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:01.189026 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:17:01.215928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:01.215891 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:17:01.518397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:01.518369 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:17:05.245049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:05.245019 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:17:05.245596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:05.245421 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:17:07.882459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.882429 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.910937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.910942 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911010 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911222 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911342 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911463 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911510 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xjh\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.911596 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:07.923630 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.012864 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.012898 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.012952 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.012982 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013008 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xjh\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013399 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013413 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013532 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013567 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013692 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.013757 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.014541 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.015488 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.015583 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.042910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xjh\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.148620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.046826 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.247379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.247352 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:08.573675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.573643 4543 generic.go:356] "Generic (PLEG): container finished" podID="c513c5dc-9256-40cf-a616-ada014206f81" containerID="41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc" exitCode=0 Oct 29 22:17:08.573863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.573720 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerDied","Data":"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc"} Oct 29 22:17:08.574190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.574171 4543 scope.go:122] "RemoveContainer" containerID="41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc" Oct 29 22:17:08.575558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.575530 4543 generic.go:356] "Generic (PLEG): container finished" podID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerID="a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7" exitCode=0 Oct 29 22:17:08.575656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.575605 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerDied","Data":"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7"} Oct 29 22:17:08.575950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.575934 4543 scope.go:122] "RemoveContainer" containerID="a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7" Oct 29 22:17:08.576029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.576001 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.576124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.576094 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.576202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.576133 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.577548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577524 4543 generic.go:356] "Generic (PLEG): container finished" podID="74f04497-2263-427b-8521-d79e20303f98" containerID="f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e" exitCode=0 Oct 29 22:17:08.577642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577589 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerDied","Data":"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e"} Oct 29 22:17:08.577806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577782 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.577859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577846 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.577895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577882 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.577933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.577903 4543 scope.go:122] "RemoveContainer" containerID="f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e" Oct 29 22:17:08.580753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.580720 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.580838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.580808 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.580883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.580868 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.589309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.589286 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:17:08.596636 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:08.596606 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ad355e_7236_4438_9ade_e4978b91f0c0.slice/crio-980bfbc61c1fa240b1a41ff352b9a5c207b337fea1647a92fdf0cd3a2e5e36b0 WatchSource:0}: Error finding container 980bfbc61c1fa240b1a41ff352b9a5c207b337fea1647a92fdf0cd3a2e5e36b0: Status 404 returned error can't find the container with id 980bfbc61c1fa240b1a41ff352b9a5c207b337fea1647a92fdf0cd3a2e5e36b0 Oct 29 22:17:08.600951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.600917 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.601047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.600986 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:08.601047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:08.601025 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:09.597587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.597549 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" event={"ID":"76ad355e-7236-4438-9ade-e4978b91f0c0","Type":"ContainerStarted","Data":"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367"} Oct 29 22:17:09.598248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.598178 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" event={"ID":"76ad355e-7236-4438-9ade-e4978b91f0c0","Type":"ContainerStarted","Data":"980bfbc61c1fa240b1a41ff352b9a5c207b337fea1647a92fdf0cd3a2e5e36b0"} Oct 29 22:17:09.600465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.600436 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerStarted","Data":"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1"} Oct 29 22:17:09.602652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.602629 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerStarted","Data":"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814"} Oct 29 22:17:09.604832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.604800 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerStarted","Data":"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac"} Oct 29 22:17:09.619640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.619488 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:17:09.622880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.621435 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:09.622880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.621496 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:09.702110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.696923 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podStartSLOduration=2.696901398 podStartE2EDuration="2.696901398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:09.664716218 +0000 UTC m=+2091.406646963" watchObservedRunningTime="2026-10-29 22:17:09.696901398 +0000 UTC m=+2091.438832072" Oct 29 22:17:09.898230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:09.898065 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:17:10.096099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.076000 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:17:10.096099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.079874 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.096099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.079928 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:10.254099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.253098 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:10.254099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.253168 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:17:10.254099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.253270 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.254099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.253312 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:10.275101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.270210 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.275101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.270283 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:10.620162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.619600 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.620162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.619674 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:10.774102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.770231 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:17:10.809496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.809457 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:10.827107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.820254 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:10.827554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.822569 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:10.836366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:10.822604 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.077033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.076490 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.077033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.076558 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:11.248536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.248495 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.248791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.248763 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:11.253416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.253388 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.253625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.253602 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:11.288861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.287148 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:17:11.527158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.464162 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.636105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.634032 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.636105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.635041 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:17:11.746103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.745682 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.746103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.745739 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:11.782157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.781881 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.807101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.782434 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:17:11.832102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.823722 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.832102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.824116 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:17:11.857101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.846164 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.857101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.846635 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:17:11.882440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.870363 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:11.882440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.881634 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:17:12.001192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.895954 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:17:12.001192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:11.896024 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:17:12.106893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.106799 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.110372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.110339 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:12.222301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.211936 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.311779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.311730 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.315740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.315714 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:17:12.339898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.336849 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.339898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.336922 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:12.483592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.483423 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.722283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.722230 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.804214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.742352 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.804214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.742577 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 29 22:17:12.820951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.820910 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.823180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.823144 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:12.823333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.821627 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:17:12.876129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.761312 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.876129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.867964 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:12.876129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.868499 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.876129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.870821 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:17:12.971102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.851273 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:12.971538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:12.971497 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:17:13.080157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.079413 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.080157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.079490 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:13.302014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.301980 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:17:13.302373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.302352 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:17:13.302549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.302537 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:17:13.353305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.307480 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.354460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.354431 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:13.354667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.344177 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.354809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.314140 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.355262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.328501 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.355432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.355411 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:13.463688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.463653 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:17:13.531104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.530188 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.716163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.665393 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.764239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.757153 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:13.874098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.809271 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.874098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.809695 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:17:13.874098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.827190 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.874098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.828382 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:13.874098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.828705 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:17:13.876611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:13.876572 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:14.042158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.001161 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:14.089313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.003370 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:17:14.091204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.091177 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:17:14.134204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.134169 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.156344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.156291 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:14.263102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.261671 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.263102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.261754 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:14.263102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.262291 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.263102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.262345 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:14.370157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.357170 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:14.550173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.484508 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:14.574454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.548755 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": context deadline exceeded" Oct 29 22:17:14.649594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.609299 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:17:14.676108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.617889 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.676405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.676367 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:14.900154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.897043 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:17:14.900154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.898204 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:17:14.921108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.904697 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:14.921108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:14.905265 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:17:15.085100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.081634 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:15.085407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.085372 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:15.097131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.092158 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:15.097131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.092512 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:17:15.259561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.258414 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:17:15.259561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.258721 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:15.259561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.258769 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:15.260061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.259966 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:15.260061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.260012 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:15.349305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.349208 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:15.468517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.468476 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:15.583716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.582487 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:17:15.583716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.583069 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:17:15.624744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.624310 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:15.624744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.624371 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:15.655043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.654997 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": context deadline exceeded" Oct 29 22:17:15.850158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.840188 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:15.938523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.931474 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:17:15.980102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.954758 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 22:17:16.028110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.982476 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:16.028110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:15.982824 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:17:16.090887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.090399 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Oct 29 22:17:16.090887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.090454 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Oct 29 22:17:16.097575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.096183 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 29 22:17:16.097575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.096521 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:17:16.241731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.235369 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:17:16.269126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.269056 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 29 22:17:16.269330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.269155 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 29 22:17:16.287325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.283686 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 29 22:17:16.287325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.283746 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 29 22:17:16.400906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.400873 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:17:16.402096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.402055 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:17:16.403140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.403124 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:17:16.404343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.404323 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:17:16.488778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.488738 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:16.642611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.642519 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 29 22:17:16.646101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.642862 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 29 22:17:16.677320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.671546 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:17:16.706015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.705793 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:17:16.708377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.708225 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.708647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.708634 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" Oct 29 22:17:16.717172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.717139 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:16.733292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.729649 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0412c3-openshift-default-dockercfg-6fj7v\"" Oct 29 22:17:16.733292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.731475 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:17:16.753346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.753041 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:17:16.833110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.823186 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.942600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxqq\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.942748 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.942795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.942837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.943038 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.943173 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.943481 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.943544 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:16.944162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:16.943651 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063093 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063252 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063295 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063348 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxqq\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063423 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063494 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.064103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.063618 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.077583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.066654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.077583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.067316 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.077583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.067645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.077583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.067861 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.077583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.068701 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.136106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.116654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.167612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.167579 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.197720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.197634 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.259629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.259598 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:17:17.277687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.276388 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:17:17.292541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.292498 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:17.294141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.292758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxqq\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\") pod \"e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.295496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.295478 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:17.301087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.301048 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:17:17.325226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.324811 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:17:17.331228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.331202 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:17:17.560252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.560218 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:17.624920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.624891 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:17:17.627832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.627810 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:17:17.945682 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:17.945649 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16723af5_2c7c_4e44_a2bd_8722d3f75bcd.slice/crio-5242b95ad6f54b14ccd0a42c8ffaa63c6754c32efefdd82a938b20effc455f7d WatchSource:0}: Error finding container 5242b95ad6f54b14ccd0a42c8ffaa63c6754c32efefdd82a938b20effc455f7d: Status 404 returned error can't find the container with id 5242b95ad6f54b14ccd0a42c8ffaa63c6754c32efefdd82a938b20effc455f7d Oct 29 22:17:17.946095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.946056 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:17:17.948900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.948865 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:17.949017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.948936 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:17.949017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:17.948968 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:18.221832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:18.221795 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerStarted","Data":"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2"} Oct 29 22:17:18.222007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:18.221861 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerStarted","Data":"5242b95ad6f54b14ccd0a42c8ffaa63c6754c32efefdd82a938b20effc455f7d"} Oct 29 22:17:18.273252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:18.273208 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podStartSLOduration=2.273193643 podStartE2EDuration="2.273193643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:18.269610667 +0000 UTC m=+2100.011541342" watchObservedRunningTime="2026-10-29 22:17:18.273193643 +0000 UTC m=+2100.015124504" Oct 29 22:17:18.566182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:18.563195 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:19.569487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:19.569455 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:20.239667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:20.239643 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:20.240458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:20.240437 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:17:22.483139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.483111 4543 scope.go:122] "RemoveContainer" containerID="4bd6a554a3f3f4df325c3be0a01b48e0f9aa7e9c77327cc2d821163b165876d5" Oct 29 22:17:22.496313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.496288 4543 scope.go:122] "RemoveContainer" containerID="e09585457b278baf15bb5aa2921ef0fd6fb44a23607cfc2a1e18a5582632c288" Oct 29 22:17:22.518875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.518846 4543 scope.go:122] "RemoveContainer" containerID="1c1861359c7fe68f7cfa210e017fc646beb3c6db7a64f00166a211b15a80f0c2" Oct 29 22:17:22.530857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.530828 4543 scope.go:122] "RemoveContainer" containerID="a97a5e93b03b3730676edbbb1b84665793fcb0fffae24178f6bd85ee0e6d2f3c" Oct 29 22:17:22.542906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.542886 4543 scope.go:122] "RemoveContainer" containerID="924c07238de41a7213745be2f7fe559187e0c8824823a1250cadcdb061856a02" Oct 29 22:17:22.552555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.552538 4543 scope.go:122] "RemoveContainer" containerID="e5659a79370489909f5ef46f3f580bc772217610c1acf0f9c4c72ed53ae35409" Oct 29 22:17:22.562236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.562220 4543 scope.go:122] "RemoveContainer" containerID="10389de19d71d65ca7cd3a8e3fc62606a655c6062b581fcdcaa0fc6fccd290af" Oct 29 22:17:22.572089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.572058 4543 scope.go:122] "RemoveContainer" containerID="b09f7cf11430ea9f623187db110de0d57632fee086b1435c5fbb32e375b1de2f" Oct 29 22:17:22.935816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:22.935647 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:17:23.064312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.064280 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:17:23.064549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.064447 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.084264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.084192 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0412c3-dockercfg-kjjld\"" Oct 29 22:17:23.085004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.084981 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:17:23.136869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.136834 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.136869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.136868 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qk62\" (UniqueName: \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.238373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.238348 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.238554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.238379 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qk62\" (UniqueName: \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.239136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.239116 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.265625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.265602 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qk62\" (UniqueName: \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\") pod \"payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.381592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.381567 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:23.689926 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:23.689893 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c96591e_dfa1_4d9b_8069_cb805f13d039.slice/crio-4f80cc2d363e77266a06ceab9858856e106f612e182acae7eba9f3c3b5698f68 WatchSource:0}: Error finding container 4f80cc2d363e77266a06ceab9858856e106f612e182acae7eba9f3c3b5698f68: Status 404 returned error can't find the container with id 4f80cc2d363e77266a06ceab9858856e106f612e182acae7eba9f3c3b5698f68 Oct 29 22:17:23.689926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:23.689917 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:17:24.291160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:24.286559 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" event={"ID":"1c96591e-dfa1-4d9b-8069-cb805f13d039","Type":"ContainerStarted","Data":"b44d6b6e33052af8c5fa721c17c5840e708fbb246fcef7bda5d008e0391d7a34"} Oct 29 22:17:24.291160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:24.286700 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" event={"ID":"1c96591e-dfa1-4d9b-8069-cb805f13d039","Type":"ContainerStarted","Data":"4f80cc2d363e77266a06ceab9858856e106f612e182acae7eba9f3c3b5698f68"} Oct 29 22:17:24.291160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:24.286996 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:24.336032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:24.335973 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" podStartSLOduration=2.335960982 podStartE2EDuration="2.335960982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:24.333866912 +0000 UTC m=+2106.075797586" watchObservedRunningTime="2026-10-29 22:17:24.335960982 +0000 UTC m=+2106.077891675" Oct 29 22:17:27.962757 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:17:27.962721 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:17:28.630837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.630803 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:17:28.631026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.631007 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" containerID="cri-o://fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6" gracePeriod=30 Oct 29 22:17:28.647039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.647004 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:17:28.647344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.647315 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" containerID="cri-o://7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778" gracePeriod=30 Oct 29 22:17:28.654767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.654739 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:17:28.654990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:28.654964 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" containerID="cri-o://0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc" gracePeriod=30 Oct 29 22:17:34.006998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.006968 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:17:34.011362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.011335 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:17:34.045240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.045212 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2"] Oct 29 22:17:34.052579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.052551 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad3ed25d-4w7m2"] Oct 29 22:17:34.080302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.080240 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:17:34.181917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.181880 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.182178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.181947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.182178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182020 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.182178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182123 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.182178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182159 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182187 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182212 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182254 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182285 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182321 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182351 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182398 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.182479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182431 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182580 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182616 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182641 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182665 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182764 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182794 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182830 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182848 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182933 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.182981 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.183017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.183792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.183098 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\") pod \"ff51eaed-e440-4001-a6dc-0f067a99d86c\" (UID: \"ff51eaed-e440-4001-a6dc-0f067a99d86c\") " Oct 29 22:17:34.183792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.183133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.183179 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\") pod \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\" (UID: \"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a\") " Oct 29 22:17:34.183792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.183288 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\") pod \"0b1af3ef-ce1a-4439-8310-b43f160815eb\" (UID: \"0b1af3ef-ce1a-4439-8310-b43f160815eb\") " Oct 29 22:17:34.185282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.185253 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "kube-api-access-5fgzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.185773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.185573 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "kube-api-access-jgcj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.186320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.186022 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:34.187410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.187103 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.187788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.187561 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:17:34.187788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.187588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.187950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.187899 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:34.188323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.188174 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.189153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.189125 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.189399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.189378 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.190310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.189524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.190416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.189589 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.190416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.189635 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.190547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190531 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.190612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190007 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190567 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190583 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190596 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190607 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190620 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190638 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190651 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgzr\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-kube-api-access-5fgzr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190666 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgcj6\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-kube-api-access-jgcj6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190679 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff51eaed-e440-4001-a6dc-0f067a99d86c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190692 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190704 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190717 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.190893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190730 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b1af3ef-ce1a-4439-8310-b43f160815eb-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.191407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.190919 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.194259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194231 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:17:34.194362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194238 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.194618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194596 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:17:34.194618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "kube-api-access-w7xg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.194771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194656 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:34.194771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.194676 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy" pod "0b1af3ef-ce1a-4439-8310-b43f160815eb" (UID: "0b1af3ef-ce1a-4439-8310-b43f160815eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.195252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.195230 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy" pod "ff51eaed-e440-4001-a6dc-0f067a99d86c" (UID: "ff51eaed-e440-4001-a6dc-0f067a99d86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.196267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.196245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy" pod "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" (UID: "e2f06d6e-8baa-46c1-b142-0ce3ebafee1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:34.293034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293002 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293031 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293046 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293057 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293066 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xg4\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-kube-api-access-w7xg4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293097 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293105 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293113 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff51eaed-e440-4001-a6dc-0f067a99d86c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293121 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293129 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293137 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293145 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.293278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.293152 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b1af3ef-ce1a-4439-8310-b43f160815eb-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:17:34.399713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.399675 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec28cfc5-145f-4945-8c66-08b96d4acb3e" path="/var/lib/kubelet/pods/ec28cfc5-145f-4945-8c66-08b96d4acb3e/volumes" Oct 29 22:17:34.409926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.409882 4543 generic.go:356] "Generic (PLEG): container finished" podID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerID="0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc" exitCode=0 Oct 29 22:17:34.410152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.409946 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" Oct 29 22:17:34.410152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.410016 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" event={"ID":"0b1af3ef-ce1a-4439-8310-b43f160815eb","Type":"ContainerDied","Data":"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc"} Oct 29 22:17:34.410152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.410126 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g" event={"ID":"0b1af3ef-ce1a-4439-8310-b43f160815eb","Type":"ContainerDied","Data":"4cf9dd597db0c30fd631d1707a03afaf4bc08e37d38f4f79e0862e29feebd434"} Oct 29 22:17:34.410351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.410159 4543 scope.go:122] "RemoveContainer" containerID="0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc" Oct 29 22:17:34.412238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.412110 4543 generic.go:356] "Generic (PLEG): container finished" podID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerID="fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6" exitCode=0 Oct 29 22:17:34.412238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.412161 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" event={"ID":"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a","Type":"ContainerDied","Data":"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6"} Oct 29 22:17:34.412238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.412173 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" Oct 29 22:17:34.412238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.412207 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv" event={"ID":"e2f06d6e-8baa-46c1-b142-0ce3ebafee1a","Type":"ContainerDied","Data":"f1d05fcff78e8b80e766ea848afc3723c315b18a92261255a0d180c7b8607bf3"} Oct 29 22:17:34.415700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.415678 4543 generic.go:356] "Generic (PLEG): container finished" podID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerID="7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778" exitCode=0 Oct 29 22:17:34.415831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.415734 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" Oct 29 22:17:34.415831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.415760 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" event={"ID":"ff51eaed-e440-4001-a6dc-0f067a99d86c","Type":"ContainerDied","Data":"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778"} Oct 29 22:17:34.415831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.415803 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw" event={"ID":"ff51eaed-e440-4001-a6dc-0f067a99d86c","Type":"ContainerDied","Data":"7dfde03c083343dff2a1378715c440159c0aade9d80572cb55f3685fbee0cad8"} Oct 29 22:17:34.424675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.424655 4543 scope.go:122] "RemoveContainer" containerID="0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc" Oct 29 22:17:34.424923 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:17:34.424905 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc\": container with ID starting with 0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc not found: ID does not exist" containerID="0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc" Oct 29 22:17:34.424972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.424931 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc"} err="failed to get container status \"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc\": rpc error: code = NotFound desc = could not find container \"0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc\": container with ID starting with 0d62b0966c1f78e2d1246b3e66bb916ddc5818f27192cfb84f20ef87587ce2fc not found: ID does not exist" Oct 29 22:17:34.424972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.424946 4543 scope.go:122] "RemoveContainer" containerID="fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6" Oct 29 22:17:34.436011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.435981 4543 scope.go:122] "RemoveContainer" containerID="fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6" Oct 29 22:17:34.436348 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:17:34.436329 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6\": container with ID starting with fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6 not found: ID does not exist" containerID="fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6" Oct 29 22:17:34.436419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.436356 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6"} err="failed to get container status \"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6\": rpc error: code = NotFound desc = could not find container \"fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6\": container with ID starting with fbbe52aaf0af1e43ddfd39c30b7f8d564b0892f84bdc004c4fe7d196f2197df6 not found: ID does not exist" Oct 29 22:17:34.436419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.436374 4543 scope.go:122] "RemoveContainer" containerID="7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778" Oct 29 22:17:34.450627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.450592 4543 scope.go:122] "RemoveContainer" containerID="7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778" Oct 29 22:17:34.450886 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:17:34.450867 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778\": container with ID starting with 7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778 not found: ID does not exist" containerID="7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778" Oct 29 22:17:34.450945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.450895 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778"} err="failed to get container status \"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778\": rpc error: code = NotFound desc = could not find container \"7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778\": container with ID starting with 7cf46db46d5d3b7314c37be5e2856fc34ae79c8c5e919bd29d7a0ce627452778 not found: ID does not exist" Oct 29 22:17:34.473182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.473161 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:17:34.477599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.477581 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-5cphv"] Oct 29 22:17:34.514873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.514851 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:17:34.521527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.521506 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-kx8tw"] Oct 29 22:17:34.559211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.559191 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:17:34.564887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:34.564865 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-x2f6g"] Oct 29 22:17:35.299948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:35.299922 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:17:36.388499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:36.388464 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" path="/var/lib/kubelet/pods/0b1af3ef-ce1a-4439-8310-b43f160815eb/volumes" Oct 29 22:17:36.389185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:36.389162 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" path="/var/lib/kubelet/pods/e2f06d6e-8baa-46c1-b142-0ce3ebafee1a/volumes" Oct 29 22:17:36.389696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:36.389676 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" path="/var/lib/kubelet/pods/ff51eaed-e440-4001-a6dc-0f067a99d86c/volumes" Oct 29 22:17:37.039516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:37.039484 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc"] Oct 29 22:17:37.051658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:37.051629 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b566f05-8k2qc"] Oct 29 22:17:38.388195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.388163 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48a0788-2494-4f53-bd4d-fd01e3da7947" path="/var/lib/kubelet/pods/a48a0788-2494-4f53-bd4d-fd01e3da7947/volumes" Oct 29 22:17:38.603181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603153 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:17:38.603913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603898 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" Oct 29 22:17:38.603974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603914 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" Oct 29 22:17:38.603974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603933 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" Oct 29 22:17:38.603974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603938 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" Oct 29 22:17:38.604165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603981 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" Oct 29 22:17:38.604165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.603986 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" Oct 29 22:17:38.604165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.604120 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff51eaed-e440-4001-a6dc-0f067a99d86c" containerName="istio-proxy" Oct 29 22:17:38.604165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.604133 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1af3ef-ce1a-4439-8310-b43f160815eb" containerName="istio-proxy" Oct 29 22:17:38.604165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.604140 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f06d6e-8baa-46c1-b142-0ce3ebafee1a" containerName="istio-proxy" Oct 29 22:17:38.621334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.621310 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.636255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.636230 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:17:38.640772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.640749 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0c3d9a-openshift-default-dockercfg-2x7hd\"" Oct 29 22:17:38.653459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653615 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653648 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653750 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653892 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.653979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.653937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.654033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.654024 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.654136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.654059 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5xr\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767330 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5xr\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767534 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767602 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767637 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767663 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767726 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.767982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.767955 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.768482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.768458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.777121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.768696 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.777121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.770817 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.777121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.771676 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.777421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.777400 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.810097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.797389 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.843132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.835194 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.896147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.895366 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.918100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.915800 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5xr\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\") pod \"e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:38.959123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:38.947435 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:39.335295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.335267 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:17:39.335669 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:39.335648 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877914d6_a5e3_4201_861f_52de61c9d9ff.slice/crio-009253fabb2bb2db7cabf3d20fb404cca99a48053625cb7fcada2ff7fe9893f8 WatchSource:0}: Error finding container 009253fabb2bb2db7cabf3d20fb404cca99a48053625cb7fcada2ff7fe9893f8: Status 404 returned error can't find the container with id 009253fabb2bb2db7cabf3d20fb404cca99a48053625cb7fcada2ff7fe9893f8 Oct 29 22:17:39.338420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.338386 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:39.338518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.338450 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:39.338518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.338479 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:17:39.462532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.462500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerStarted","Data":"7f3ae20bebfb73974c30d35f178ef3232709feddae783fd00a67f4f1225a38c2"} Oct 29 22:17:39.463010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.462550 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerStarted","Data":"009253fabb2bb2db7cabf3d20fb404cca99a48053625cb7fcada2ff7fe9893f8"} Oct 29 22:17:39.513304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.513258 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podStartSLOduration=1.5132435640000002 podStartE2EDuration="1.513243564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:39.507645162 +0000 UTC m=+2121.249575848" watchObservedRunningTime="2026-10-29 22:17:39.513243564 +0000 UTC m=+2121.255174236" Oct 29 22:17:39.950986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:39.950953 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:40.956758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:40.956725 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:41.480632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:41.480600 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:41.481451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:41.481428 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:17:43.986900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:43.986866 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:17:43.990547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:43.990531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.012092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.012054 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:17:44.020801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.020765 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0c3d9a-dockercfg-9phkt\"" Oct 29 22:17:44.020911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.020840 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0c3d9a\"" Oct 29 22:17:44.148781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.148737 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8j7\" (UniqueName: \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.149253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.149223 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.183226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.183193 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:17:44.300214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.282542 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.300214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.282897 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8j7\" (UniqueName: \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.300214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.284189 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.329239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.317682 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.329239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.326358 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:17:44.335243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.335195 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8j7\" (UniqueName: \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\") pod \"payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.384841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.384794 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5vh\" (UniqueName: \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.385012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.384946 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.486300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.486260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5vh\" (UniqueName: \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.486482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.486458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.487169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.487150 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.510777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.510752 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5vh\" (UniqueName: \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\") pod \"payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.605042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.604980 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:44.720090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.720050 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:44.914604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:44.914580 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:17:44.915362 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:44.915336 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b9d7c8_9ed4_4c74_a9c2_0bf1f638769f.slice/crio-28f56f6ea322094b01fd6a842df334e6c66bf75b6d92772cd639c2800b2177dc WatchSource:0}: Error finding container 28f56f6ea322094b01fd6a842df334e6c66bf75b6d92772cd639c2800b2177dc: Status 404 returned error can't find the container with id 28f56f6ea322094b01fd6a842df334e6c66bf75b6d92772cd639c2800b2177dc Oct 29 22:17:45.049330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.049304 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:17:45.049863 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:17:45.049836 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb15e08_2fff_48d7_87f0_1b0f6de25ba3.slice/crio-af7403afde46442df15cdcd28f3fd35a759b9ce973a9ece2899ab9768996c4d2 WatchSource:0}: Error finding container af7403afde46442df15cdcd28f3fd35a759b9ce973a9ece2899ab9768996c4d2: Status 404 returned error can't find the container with id af7403afde46442df15cdcd28f3fd35a759b9ce973a9ece2899ab9768996c4d2 Oct 29 22:17:45.563835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.563801 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" event={"ID":"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3","Type":"ContainerStarted","Data":"0535c052e3c02735eff13830054e5f5c68ab1bebf98b0bd795911aba5e67322c"} Oct 29 22:17:45.564003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.563858 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" event={"ID":"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3","Type":"ContainerStarted","Data":"af7403afde46442df15cdcd28f3fd35a759b9ce973a9ece2899ab9768996c4d2"} Oct 29 22:17:45.564003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.563881 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:45.565733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.565699 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" event={"ID":"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f","Type":"ContainerStarted","Data":"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b"} Oct 29 22:17:45.565894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.565747 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" event={"ID":"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f","Type":"ContainerStarted","Data":"28f56f6ea322094b01fd6a842df334e6c66bf75b6d92772cd639c2800b2177dc"} Oct 29 22:17:45.565894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.565788 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:45.607938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.607899 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" podStartSLOduration=1.607886708 podStartE2EDuration="1.607886708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:45.604365158 +0000 UTC m=+2127.346295835" watchObservedRunningTime="2026-10-29 22:17:45.607886708 +0000 UTC m=+2127.349817449" Oct 29 22:17:45.647919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:45.647874 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" podStartSLOduration=2.6478614 podStartE2EDuration="2.6478614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:45.645024627 +0000 UTC m=+2127.386955301" watchObservedRunningTime="2026-10-29 22:17:45.6478614 +0000 UTC m=+2127.389792076" Oct 29 22:17:50.040555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:50.040524 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7"] Oct 29 22:17:50.051990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:50.051965 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0e78e-fgft7"] Oct 29 22:17:50.387695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:50.387623 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb48345-5272-4cde-8476-7d41edc3e788" path="/var/lib/kubelet/pods/fcb48345-5272-4cde-8476-7d41edc3e788/volumes" Oct 29 22:17:56.579375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:56.579345 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:17:56.579925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:17:56.579408 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:17:57.962047 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:17:57.962004 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:18:14.024458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.024415 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:18:14.029667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.029640 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.033776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.032966 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:18:14.049507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.049404 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7d8760-openshift-default-dockercfg-sgbz8\"" Oct 29 22:18:14.174358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.174320 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.174739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.174717 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.174968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.174950 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsx2\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.175216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.175199 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.175395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.175379 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.191232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.191190 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.191589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.191562 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.191888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.191862 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.192086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.192053 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsx2\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294290 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294361 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294407 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294434 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294456 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294478 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294534 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294695 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.294815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294785 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.295038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294859 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.295038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.294884 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.295364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.295342 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.297710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.297687 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.298008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.297985 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.318428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.318407 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.318687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.318665 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsx2\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\") pod \"e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.381327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.381291 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:14.781028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.781003 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:18:14.785735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.785692 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:14.785858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.785768 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:14.785858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.785798 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:14.972972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.972935 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerStarted","Data":"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a"} Oct 29 22:18:14.973206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:14.972996 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerStarted","Data":"4e2c95aedac732981ac6f0d2e8f5a6d5f7507232697a40eb2518b2bebd3a1636"} Oct 29 22:18:15.027998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:15.027951 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podStartSLOduration=2.027936646 podStartE2EDuration="2.027936646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:15.024237354 +0000 UTC m=+2156.766168028" watchObservedRunningTime="2026-10-29 22:18:15.027936646 +0000 UTC m=+2156.769867322" Oct 29 22:18:15.381884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:15.381843 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:15.382910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:15.382876 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:18:15.383046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:15.382931 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:18:16.389572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:16.389545 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:17.068098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:17.067152 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:17.073058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:17.071374 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:18:19.609254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.609219 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:18:19.615262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.615240 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.634044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.634014 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:18:19.643125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.640979 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7d8760\"" Oct 29 22:18:19.643125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.640989 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7d8760-dockercfg-przjh\"" Oct 29 22:18:19.799527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.742163 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtms\" (UniqueName: \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.807104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.800024 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.824107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.764130 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:18:19.905254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.903606 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:18:19.905254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.903733 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:19.941161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.914641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.941161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.914964 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtms\" (UniqueName: \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.941161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.916645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:19.949388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:19.949323 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtms\" (UniqueName: \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\") pod \"payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:20.018518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.017624 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfgn\" (UniqueName: \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.018518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.017752 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.119218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.119052 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfgn\" (UniqueName: \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.119218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.119194 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.120175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.120149 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.143894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.143860 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfgn\" (UniqueName: \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\") pod \"payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.233248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.233219 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:20.309407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.309030 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:20.556896 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:18:20.556864 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2036a9c_816c_45bf_b4d8_dc59815ff9f4.slice/crio-4f6d4119cf14f3a06152e71f01fd3c8226c91d890d271b49ada92e107ca24706 WatchSource:0}: Error finding container 4f6d4119cf14f3a06152e71f01fd3c8226c91d890d271b49ada92e107ca24706: Status 404 returned error can't find the container with id 4f6d4119cf14f3a06152e71f01fd3c8226c91d890d271b49ada92e107ca24706 Oct 29 22:18:20.557008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.556945 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:18:20.671166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:20.669978 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:18:20.673544 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:18:20.672567 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb450c07e_fd5b_486f_9f9e_d5e299b3aad5.slice/crio-f22e74bee3e35d678478e831b1222ce38bdada11f26b65e4774a2063058de48d WatchSource:0}: Error finding container f22e74bee3e35d678478e831b1222ce38bdada11f26b65e4774a2063058de48d: Status 404 returned error can't find the container with id f22e74bee3e35d678478e831b1222ce38bdada11f26b65e4774a2063058de48d Oct 29 22:18:21.109763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.109722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" event={"ID":"b450c07e-fd5b-486f-9f9e-d5e299b3aad5","Type":"ContainerStarted","Data":"fb7bc2b854462de810345e048b4588ac5d84a6cb23ddd64d822ec778e13e4779"} Oct 29 22:18:21.109983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.109820 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" event={"ID":"b450c07e-fd5b-486f-9f9e-d5e299b3aad5","Type":"ContainerStarted","Data":"f22e74bee3e35d678478e831b1222ce38bdada11f26b65e4774a2063058de48d"} Oct 29 22:18:21.110062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.110042 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:21.111837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.111809 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" event={"ID":"d2036a9c-816c-45bf-b4d8-dc59815ff9f4","Type":"ContainerStarted","Data":"09b517d0c4ca8a301faebc3505b4f8e764db07ff63ed91d30febe49cde806a87"} Oct 29 22:18:21.111970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.111858 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" event={"ID":"d2036a9c-816c-45bf-b4d8-dc59815ff9f4","Type":"ContainerStarted","Data":"4f6d4119cf14f3a06152e71f01fd3c8226c91d890d271b49ada92e107ca24706"} Oct 29 22:18:21.111970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.111890 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:21.158116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.158001 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" podStartSLOduration=2.157984938 podStartE2EDuration="2.157984938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:21.151756304 +0000 UTC m=+2162.893686978" watchObservedRunningTime="2026-10-29 22:18:21.157984938 +0000 UTC m=+2162.899915612" Oct 29 22:18:21.197415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:21.197359 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" podStartSLOduration=2.197344805 podStartE2EDuration="2.197344805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:21.193574982 +0000 UTC m=+2162.935505655" watchObservedRunningTime="2026-10-29 22:18:21.197344805 +0000 UTC m=+2162.939275477" Oct 29 22:18:22.654442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:22.654407 4543 scope.go:122] "RemoveContainer" containerID="e0eb44e6b228b10465a97607658713de30ec68acdf2cf347b2ad2e7ad9634040" Oct 29 22:18:22.668302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:22.668272 4543 scope.go:122] "RemoveContainer" containerID="7a79610016ccad20ca01dc8289c85efe00badd72330d4bedf17c99d8453982f5" Oct 29 22:18:22.679207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:22.679184 4543 scope.go:122] "RemoveContainer" containerID="2748f6945d9592c74bc9039787c3517b603ca71fd81e9632d6105bcecde71e90" Oct 29 22:18:22.691430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:22.691402 4543 scope.go:122] "RemoveContainer" containerID="1a824072c231b773f35e6b3215d38ba4fdbeac57d005edccabc7069368506a62" Oct 29 22:18:22.703044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:22.703024 4543 scope.go:122] "RemoveContainer" containerID="000d2b4ecb79961d75d783b9ce99332b49689baf21556592b1f48e6a8e5365ce" Oct 29 22:18:24.140366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.140333 4543 generic.go:356] "Generic (PLEG): container finished" podID="9560df2f-92d0-474d-b279-283cb493c35c" containerID="8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0" exitCode=0 Oct 29 22:18:24.140812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.140395 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerDied","Data":"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0"} Oct 29 22:18:24.140922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.140902 4543 scope.go:122] "RemoveContainer" containerID="8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0" Oct 29 22:18:24.142728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.142696 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:24.142833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.142769 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:24.142833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:24.142802 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:25.162684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:25.162652 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerStarted","Data":"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882"} Oct 29 22:18:26.087669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:26.087637 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:18:26.087916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:26.087894 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:18:26.087980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:26.087951 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:18:27.088234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:27.088201 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:18:27.088618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:27.088264 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:18:27.987799 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:18:27.987762 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:18:28.088733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:28.088703 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:18:28.089141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:28.088766 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:18:29.088625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:29.088593 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:18:29.088803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:29.088645 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:18:30.088446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:30.088410 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Oct 29 22:18:30.088664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:30.088469 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Oct 29 22:18:31.093806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:31.093774 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:18:31.094212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:31.094164 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:18:31.094847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:31.094828 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:18:32.124808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:32.124778 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:18:32.125237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:32.124858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:18:38.519771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:38.519737 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 22:18:38.520210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:38.519908 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" containerID="cri-o://d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f" gracePeriod=30 Oct 29 22:18:43.039265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.039229 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2"] Oct 29 22:18:43.050780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.050747 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e03a25d-k76j2"] Oct 29 22:18:43.530418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.530061 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:18:43.530418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.530148 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 29 22:18:43.823961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.823935 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 22:18:43.916367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916338 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916392 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916455 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916560 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916670 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916823 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.916938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916885 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.917064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.916984 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\") pod \"9987146b-2f54-4aba-8550-8b9b55e2e00c\" (UID: \"9987146b-2f54-4aba-8550-8b9b55e2e00c\") " Oct 29 22:18:43.918333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.917430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:43.918333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.917442 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:18:43.918333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.917610 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:43.918333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.917665 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:43.918333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.917687 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:43.918695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.918452 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:43.918695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.918474 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:43.918695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.918489 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:43.918695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.918503 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9987146b-2f54-4aba-8550-8b9b55e2e00c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:43.918695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.918517 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:43.919375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.919336 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "kube-api-access-qkzwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:43.919509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.919438 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:18:43.919777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.919756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:18:43.920557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:43.920539 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token" pod "9987146b-2f54-4aba-8550-8b9b55e2e00c" (UID: "9987146b-2f54-4aba-8550-8b9b55e2e00c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:18:44.020124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.020092 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:44.020124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.020120 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzwj\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-kube-api-access-qkzwj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:44.020124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.020132 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:44.020377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.020141 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9987146b-2f54-4aba-8550-8b9b55e2e00c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:18:44.360837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.360794 4543 generic.go:356] "Generic (PLEG): container finished" podID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerID="d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f" exitCode=0 Oct 29 22:18:44.361368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.360860 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" Oct 29 22:18:44.361368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.360873 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" event={"ID":"9987146b-2f54-4aba-8550-8b9b55e2e00c","Type":"ContainerDied","Data":"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f"} Oct 29 22:18:44.361368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.360934 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n" event={"ID":"9987146b-2f54-4aba-8550-8b9b55e2e00c","Type":"ContainerDied","Data":"a2847c75d91fb73d07eb94359d38754c692b9ea44ce20b27e8e1935297153c24"} Oct 29 22:18:44.361368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.360980 4543 scope.go:122] "RemoveContainer" containerID="d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f" Oct 29 22:18:44.373779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.373763 4543 scope.go:122] "RemoveContainer" containerID="d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f" Oct 29 22:18:44.374060 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:18:44.374039 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f\": container with ID starting with d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f not found: ID does not exist" containerID="d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f" Oct 29 22:18:44.374153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.374069 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f"} err="failed to get container status \"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f\": rpc error: code = NotFound desc = could not find container \"d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f\": container with ID starting with d92cbe7f6e3daaa911e881951fd47f9898e0322b5120f85909162817e496881f not found: ID does not exist" Oct 29 22:18:44.388908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.388884 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efbb881-d26c-473b-983b-5e208e00c104" path="/var/lib/kubelet/pods/6efbb881-d26c-473b-983b-5e208e00c104/volumes" Oct 29 22:18:44.417256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.417233 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 22:18:44.429251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.429228 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-rzh2n"] Oct 29 22:18:44.709981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.709902 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:18:44.717629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.711143 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" Oct 29 22:18:44.717629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.711169 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" Oct 29 22:18:44.717629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.711480 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" containerName="istio-proxy" Oct 29 22:18:44.717629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.717491 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.740900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.740869 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e7b56d-openshift-default-dockercfg-p6kdk\"" Oct 29 22:18:44.748257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.748233 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842377 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszh9\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842529 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842566 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842665 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842735 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.842886 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.843000 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:44.849099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:44.843037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025331 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025394 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025500 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszh9\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025623 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025652 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025805 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.025948 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.034499 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.034781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.034758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.052100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.035384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.052100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.035633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.052100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.036558 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.052100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.038935 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.080103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.065653 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.080103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.072008 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszh9\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.091098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.089703 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\") pod \"e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.349839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.349760 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:45.707958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.707928 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:18:45.708287 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:18:45.707934 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f7ed30_5a2e_417d_9c60_cdf53104ef89.slice/crio-dcd613599788bda0c1dc4d680498fe3b585c48f7c78ed7f1e7e04e439636209d WatchSource:0}: Error finding container dcd613599788bda0c1dc4d680498fe3b585c48f7c78ed7f1e7e04e439636209d: Status 404 returned error can't find the container with id dcd613599788bda0c1dc4d680498fe3b585c48f7c78ed7f1e7e04e439636209d Oct 29 22:18:45.710909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.710876 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:45.711017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.710939 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:45.711017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:45.710969 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:18:46.389056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:46.389011 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9987146b-2f54-4aba-8550-8b9b55e2e00c" path="/var/lib/kubelet/pods/9987146b-2f54-4aba-8550-8b9b55e2e00c/volumes" Oct 29 22:18:46.398457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:46.398426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerStarted","Data":"7be5bba1238928e4b663decfda0fb4698bae106748324f61f56936ef1e1ae11e"} Oct 29 22:18:46.398616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:46.398485 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerStarted","Data":"dcd613599788bda0c1dc4d680498fe3b585c48f7c78ed7f1e7e04e439636209d"} Oct 29 22:18:46.446832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:46.446788 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podStartSLOduration=2.446776082 podStartE2EDuration="2.446776082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:46.444252277 +0000 UTC m=+2188.186182949" watchObservedRunningTime="2026-10-29 22:18:46.446776082 +0000 UTC m=+2188.188706755" Oct 29 22:18:47.350166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:47.350134 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:47.356024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:47.355999 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:47.408875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:47.408843 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:47.409689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:47.409668 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:18:50.864350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.864315 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:18:50.870404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.870385 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:50.890219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.890195 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:18:50.896992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.896963 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e7b56d\"" Oct 29 22:18:50.897205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.897157 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e7b56d-dockercfg-5mdwp\"" Oct 29 22:18:50.912814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.912787 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:50.913049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:50.913029 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkk4q\" (UniqueName: \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.120153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.087132 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:18:51.135348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.134369 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.164122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.138935 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:18:51.174094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.174043 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.176029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.175999 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkk4q\" (UniqueName: \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.176800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.175304 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.243103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.242874 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkk4q\" (UniqueName: \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\") pod \"payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.279224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.278823 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsp2\" (UniqueName: \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.279224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.278909 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.380423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.380335 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsp2\" (UniqueName: \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.380423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.380401 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.381065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.381043 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.405615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.405585 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsp2\" (UniqueName: \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\") pod \"payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.485694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.485673 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:51.584835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.584795 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:51.807401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.806368 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:18:51.809568 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:18:51.809537 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93e4d70_8859_441f_ab72_8d9b660b02cb.slice/crio-8a4630c3c756f8ee76e062a2fbd44e2bfa0dfc2c2ff6bf764dcbb34240efd7e0 WatchSource:0}: Error finding container 8a4630c3c756f8ee76e062a2fbd44e2bfa0dfc2c2ff6bf764dcbb34240efd7e0: Status 404 returned error can't find the container with id 8a4630c3c756f8ee76e062a2fbd44e2bfa0dfc2c2ff6bf764dcbb34240efd7e0 Oct 29 22:18:51.947131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:51.947045 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:18:51.947131 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:18:51.947051 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ad99d7_332f_417e_b1a1_888db568f7a6.slice/crio-fe61d92164da154c726741979fce4b931f226d44d14e8ca82f9271618fa64726 WatchSource:0}: Error finding container fe61d92164da154c726741979fce4b931f226d44d14e8ca82f9271618fa64726: Status 404 returned error can't find the container with id fe61d92164da154c726741979fce4b931f226d44d14e8ca82f9271618fa64726 Oct 29 22:18:52.455707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.455669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" event={"ID":"e93e4d70-8859-441f-ab72-8d9b660b02cb","Type":"ContainerStarted","Data":"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a"} Oct 29 22:18:52.455942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.455734 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:18:52.455942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.455759 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" event={"ID":"e93e4d70-8859-441f-ab72-8d9b660b02cb","Type":"ContainerStarted","Data":"8a4630c3c756f8ee76e062a2fbd44e2bfa0dfc2c2ff6bf764dcbb34240efd7e0"} Oct 29 22:18:52.457845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.457818 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" event={"ID":"87ad99d7-332f-417e-b1a1-888db568f7a6","Type":"ContainerStarted","Data":"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8"} Oct 29 22:18:52.457992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.457866 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" event={"ID":"87ad99d7-332f-417e-b1a1-888db568f7a6","Type":"ContainerStarted","Data":"fe61d92164da154c726741979fce4b931f226d44d14e8ca82f9271618fa64726"} Oct 29 22:18:52.457992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.457935 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:18:52.500889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.500848 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" podStartSLOduration=2.500834951 podStartE2EDuration="2.500834951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:52.495861777 +0000 UTC m=+2194.237792450" watchObservedRunningTime="2026-10-29 22:18:52.500834951 +0000 UTC m=+2194.242765621" Oct 29 22:18:52.540045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:52.539999 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" podStartSLOduration=2.539982264 podStartE2EDuration="2.539982264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:52.537183537 +0000 UTC m=+2194.279114233" watchObservedRunningTime="2026-10-29 22:18:52.539982264 +0000 UTC m=+2194.281912939" Oct 29 22:18:53.035633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:53.035600 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh"] Oct 29 22:18:53.042387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:53.042365 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a3b57f-6mhgh"] Oct 29 22:18:54.388946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:18:54.388914 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eed4bef-76c4-41d7-9fbc-7e69acc82387" path="/var/lib/kubelet/pods/5eed4bef-76c4-41d7-9fbc-7e69acc82387/volumes" Oct 29 22:18:57.963605 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:18:57.963568 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:19:03.502154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:03.499629 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:19:03.502642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:03.502272 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:19:10.052125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:10.052085 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb"] Oct 29 22:19:10.056451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:10.056425 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ec66e5e-rxltb"] Oct 29 22:19:10.392128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:10.392033 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b60aede-e716-4ae0-baac-759cc76a4e32" path="/var/lib/kubelet/pods/9b60aede-e716-4ae0-baac-759cc76a4e32/volumes" Oct 29 22:19:12.524244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:12.524209 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:19:12.524742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:12.524279 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:19:14.889625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.889586 4543 generic.go:356] "Generic (PLEG): container finished" podID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerID="7f3ae20bebfb73974c30d35f178ef3232709feddae783fd00a67f4f1225a38c2" exitCode=0 Oct 29 22:19:14.890167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.889673 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerDied","Data":"7f3ae20bebfb73974c30d35f178ef3232709feddae783fd00a67f4f1225a38c2"} Oct 29 22:19:14.890336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.890311 4543 scope.go:122] "RemoveContainer" containerID="7f3ae20bebfb73974c30d35f178ef3232709feddae783fd00a67f4f1225a38c2" Oct 29 22:19:14.891837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.891813 4543 generic.go:356] "Generic (PLEG): container finished" podID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerID="afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2" exitCode=0 Oct 29 22:19:14.891930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.891842 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerDied","Data":"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2"} Oct 29 22:19:14.892316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.892299 4543 scope.go:122] "RemoveContainer" containerID="afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2" Oct 29 22:19:14.892730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.892699 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:14.892810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.892791 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:14.892861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.892844 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:14.894346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.894316 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:14.894452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.894393 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:14.894452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:14.894428 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:15.908897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:15.908862 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerStarted","Data":"3ead69d8712c586286ed67f61555d4da9f984abb9d4c1462da26878622149e4b"} Oct 29 22:19:15.910989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:15.910960 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerStarted","Data":"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf"} Oct 29 22:19:15.961263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:15.961228 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:19:16.579006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:16.578772 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:19:16.580574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:16.580175 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" start-of-body= Oct 29 22:19:16.580574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:16.580227 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" Oct 29 22:19:16.951498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:16.951408 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:19:16.961180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:16.961128 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:19:17.191581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.188154 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:19:17.194285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.194258 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.195015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.194988 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:19:17.233332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.233286 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-07e166\"" Oct 29 22:19:17.349123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.261548 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-07e166-dockercfg-cbtbg\"" Oct 29 22:19:17.349123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.338121 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:19:17.403191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.398379 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.438632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.434601 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:19:17.438632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.438527 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.438998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.438692 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22p8\" (UniqueName: \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.438998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.438740 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.438998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.438929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwv9g\" (UniqueName: \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.550626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.550537 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22p8\" (UniqueName: \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.550890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.550869 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.553848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.551218 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwv9g\" (UniqueName: \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.629250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.629210 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.663996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.586094 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.677294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.677263 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" start-of-body= Oct 29 22:19:17.682043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.682013 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" Oct 29 22:19:17.682618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.681947 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.707126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.706758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwv9g\" (UniqueName: \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\") pod \"payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:17.707126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.707049 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22p8\" (UniqueName: \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\") pod \"payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.819442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.819352 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:17.964641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.964184 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:19:17.964641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.964247 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:19:17.996614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:17.995831 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:18.406236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.404270 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:19:18.585786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.585730 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" start-of-body= Oct 29 22:19:18.586050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.586004 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" Oct 29 22:19:18.625545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.625520 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:19:18.964163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.953573 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:19:18.964163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.953627 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:19:18.993507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.993460 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" event={"ID":"983248db-dedd-4c0e-9750-e4fc85e79e0b","Type":"ContainerStarted","Data":"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0"} Oct 29 22:19:18.993970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.993535 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" event={"ID":"983248db-dedd-4c0e-9750-e4fc85e79e0b","Type":"ContainerStarted","Data":"71a7885ee27ca4fdab8c4a6ac7d75121dbe9ba27da5cbc4e9d2ffae6af01f6d3"} Oct 29 22:19:18.994163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.994141 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:18.996620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.996596 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" event={"ID":"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b","Type":"ContainerStarted","Data":"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965"} Oct 29 22:19:18.996758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.996634 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" event={"ID":"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b","Type":"ContainerStarted","Data":"db64e36bce6396d7f5042253c997764ed460ae29057bb4015978340518250129"} Oct 29 22:19:18.996979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:18.996964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:19.083368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.083313 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" podStartSLOduration=2.0833001429999998 podStartE2EDuration="2.083300143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:19.042120459 +0000 UTC m=+2220.784051136" watchObservedRunningTime="2026-10-29 22:19:19.083300143 +0000 UTC m=+2220.825230816" Oct 29 22:19:19.096390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.096335 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" podStartSLOduration=2.096322468 podStartE2EDuration="2.096322468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:19.092783184 +0000 UTC m=+2220.834713859" watchObservedRunningTime="2026-10-29 22:19:19.096322468 +0000 UTC m=+2220.838253156" Oct 29 22:19:19.579791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.579756 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" start-of-body= Oct 29 22:19:19.580112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.580057 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" Oct 29 22:19:19.961171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.956460 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:19:19.961171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:19.956522 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:19:20.250157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:20.241701 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:19:20.572162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:20.564853 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" start-of-body= Oct 29 22:19:20.572162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:20.564931 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.54:15021/healthz/ready\": dial tcp 10.133.1.54:15021: connect: connection refused" Oct 29 22:19:20.955051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:20.954962 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 29 22:19:20.955338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:20.955310 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 29 22:19:21.627273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:21.627240 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:19:21.630967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:21.630943 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:19:21.956173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:21.956058 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:19:21.956583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:21.956564 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:19:21.957326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:21.957305 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:19:22.760215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.760185 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:19:22.760708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.760376 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" containerID="cri-o://fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e" gracePeriod=30 Oct 29 22:19:22.781218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.781188 4543 scope.go:122] "RemoveContainer" containerID="5bb362c913cae881413e93cce2d333f9127f8f4a693a601c6ccf8e3cf63d2461" Oct 29 22:19:22.794057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.794038 4543 scope.go:122] "RemoveContainer" containerID="11241233380ef32ebac3d39a613f8bb7171a0a7e6dced195178e9d6b34e86df8" Oct 29 22:19:22.812742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.811995 4543 scope.go:122] "RemoveContainer" containerID="3bac6dd66600b941d29720fd3b418a3e3d02fba9b8f58a5601f23539a153abd0" Oct 29 22:19:22.826429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.826397 4543 scope.go:122] "RemoveContainer" containerID="c4ceeea9af6bf608424137c7d3e24d5f74b2887b471490f8084a84c80e9260e9" Oct 29 22:19:22.836276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.836259 4543 scope.go:122] "RemoveContainer" containerID="8208c8813e6c0ff3c10fd60827ff175916ead06b14231555550ce08dd6200b90" Oct 29 22:19:22.845682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:22.845659 4543 scope.go:122] "RemoveContainer" containerID="eeb1ffeefdeb4d53e83092580310165d3e73fe416bfa13dd8a64b25c82d16e4b" Oct 29 22:19:27.978017 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:19:27.977959 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:19:28.063350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.063320 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:19:28.079815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.079792 4543 generic.go:356] "Generic (PLEG): container finished" podID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerID="fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e" exitCode=0 Oct 29 22:19:28.079947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.079832 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" Oct 29 22:19:28.079947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.079877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" event={"ID":"178fc4c0-85ee-4f87-932d-8a51e96cb0cf","Type":"ContainerDied","Data":"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e"} Oct 29 22:19:28.079947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.079939 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b" event={"ID":"178fc4c0-85ee-4f87-932d-8a51e96cb0cf","Type":"ContainerDied","Data":"61907b1b031bf7a2941313914c571aaeacf374afde1ee709442d33ac5be2fa18"} Oct 29 22:19:28.080179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.079971 4543 scope.go:122] "RemoveContainer" containerID="fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e" Oct 29 22:19:28.091257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.091224 4543 scope.go:122] "RemoveContainer" containerID="fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e" Oct 29 22:19:28.091537 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:19:28.091518 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e\": container with ID starting with fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e not found: ID does not exist" containerID="fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e" Oct 29 22:19:28.091626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.091549 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e"} err="failed to get container status \"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e\": rpc error: code = NotFound desc = could not find container \"fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e\": container with ID starting with fd4fb736a8643bbb2870f2aef6d2fa24c5e26592c11a01cd81cca79f8b4b249e not found: ID does not exist" Oct 29 22:19:28.145461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.145427 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.145641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.145474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146181 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.146436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146301 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146341 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146616 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146691 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146734 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.146970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.146781 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\") pod \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\" (UID: \"178fc4c0-85ee-4f87-932d-8a51e96cb0cf\") " Oct 29 22:19:28.147616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.147592 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.148460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.148037 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:28.148460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.148203 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.149308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.149279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.149483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.149461 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.151237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.151010 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "kube-api-access-4dqw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.151986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.151944 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:28.154187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.154154 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.154295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.154245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy" pod "178fc4c0-85ee-4f87-932d-8a51e96cb0cf" (UID: "178fc4c0-85ee-4f87-932d-8a51e96cb0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.248778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248751 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248775 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248785 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248793 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248801 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248808 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248823 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.248959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.248833 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqw5\" (UniqueName: \"kubernetes.io/projected/178fc4c0-85ee-4f87-932d-8a51e96cb0cf-kube-api-access-4dqw5\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.434847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.434814 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:19:28.439727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:28.439689 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-xkx6b"] Oct 29 22:19:30.053296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:30.053263 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:19:30.056878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:30.056844 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:19:30.389917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:30.389832 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" path="/var/lib/kubelet/pods/178fc4c0-85ee-4f87-932d-8a51e96cb0cf/volumes" Oct 29 22:19:34.257363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:34.257326 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:19:34.257890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:34.257595 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" containerID="cri-o://380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff" gracePeriod=30 Oct 29 22:19:34.268114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:34.268090 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:19:34.268305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:34.268284 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" containerID="cri-o://74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049" gracePeriod=30 Oct 29 22:19:39.812221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.805566 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:19:39.951784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.951688 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.952861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.952835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.953094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.953057 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.953314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.953295 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.955350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.955327 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.955620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.955602 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.978217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.978176 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.978589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.978564 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.978775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.978757 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\") pod \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\" (UID: \"957524f0-a0ac-49fa-9dfd-50ff2fa43a52\") " Oct 29 22:19:39.986614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.953465 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.986614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.984125 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:39.986614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.984344 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.002544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:39.990412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:40.002544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.002279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.002544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.002385 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:40.002544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.002501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "kube-api-access-n7zjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:40.003682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.003656 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:40.004046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.004013 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket" pod "957524f0-a0ac-49fa-9dfd-50ff2fa43a52" (UID: "957524f0-a0ac-49fa-9dfd-50ff2fa43a52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.024320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.024295 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:19:40.080760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.080729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.080973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.080864 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.080973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.080960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.080991 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081039 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081110 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081192 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081190 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:40.081318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081270 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\") pod \"699c48b1-116c-4902-9025-0505203814fd\" (UID: \"699c48b1-116c-4902-9025-0505203814fd\") " Oct 29 22:19:40.081536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081390 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.081536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081400 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.081654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081599 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.081714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.081697 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.082397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082367 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082398 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zjw\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-kube-api-access-n7zjw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082412 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082425 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082440 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082452 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082466 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082479 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082492 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082506 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082518 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.082531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082531 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.083219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082544 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957524f0-a0ac-49fa-9dfd-50ff2fa43a52-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.083219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.082557 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699c48b1-116c-4902-9025-0505203814fd-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.083442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.083411 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:40.083678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.083655 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "kube-api-access-2lzvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:40.083885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.083867 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:40.084043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.084027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token" pod "699c48b1-116c-4902-9025-0505203814fd" (UID: "699c48b1-116c-4902-9025-0505203814fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:40.184524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.184481 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.184524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.184520 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lzvv\" (UniqueName: \"kubernetes.io/projected/699c48b1-116c-4902-9025-0505203814fd-kube-api-access-2lzvv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.184524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.184531 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699c48b1-116c-4902-9025-0505203814fd-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.184925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.184543 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699c48b1-116c-4902-9025-0505203814fd-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:40.276130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.275602 4543 generic.go:356] "Generic (PLEG): container finished" podID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerID="380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff" exitCode=0 Oct 29 22:19:40.276130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.275674 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" event={"ID":"957524f0-a0ac-49fa-9dfd-50ff2fa43a52","Type":"ContainerDied","Data":"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff"} Oct 29 22:19:40.276130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.275737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" event={"ID":"957524f0-a0ac-49fa-9dfd-50ff2fa43a52","Type":"ContainerDied","Data":"3355540f6b57c58b336307c4b0dd4e669855bc1d971c917278a8beb3f7175d29"} Oct 29 22:19:40.276130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.275757 4543 scope.go:122] "RemoveContainer" containerID="380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff" Oct 29 22:19:40.276130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.275867 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75" Oct 29 22:19:40.284358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.284333 4543 generic.go:356] "Generic (PLEG): container finished" podID="699c48b1-116c-4902-9025-0505203814fd" containerID="74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049" exitCode=0 Oct 29 22:19:40.284566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.284540 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" Oct 29 22:19:40.284776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.284522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" event={"ID":"699c48b1-116c-4902-9025-0505203814fd","Type":"ContainerDied","Data":"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049"} Oct 29 22:19:40.284932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.284915 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr" event={"ID":"699c48b1-116c-4902-9025-0505203814fd","Type":"ContainerDied","Data":"db42a11bfcaecc3823292dd2defb4679661cd76ae5cfc1e86c0b0c5dba6acc74"} Oct 29 22:19:40.309495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.309478 4543 scope.go:122] "RemoveContainer" containerID="380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff" Oct 29 22:19:40.309759 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:19:40.309734 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff\": container with ID starting with 380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff not found: ID does not exist" containerID="380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff" Oct 29 22:19:40.309832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.309765 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff"} err="failed to get container status \"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff\": rpc error: code = NotFound desc = could not find container \"380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff\": container with ID starting with 380f08127b21f2947bce53431c5711a57910db255bc21d3203c4782c6645d1ff not found: ID does not exist" Oct 29 22:19:40.309832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.309782 4543 scope.go:122] "RemoveContainer" containerID="74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049" Oct 29 22:19:40.320270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.320253 4543 scope.go:122] "RemoveContainer" containerID="74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049" Oct 29 22:19:40.320517 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:19:40.320497 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049\": container with ID starting with 74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049 not found: ID does not exist" containerID="74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049" Oct 29 22:19:40.320566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.320524 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049"} err="failed to get container status \"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049\": rpc error: code = NotFound desc = could not find container \"74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049\": container with ID starting with 74600a96e1529a3e1f94b371a6e1efb4c5dec813980bf2e1b61af5f2dca80049 not found: ID does not exist" Oct 29 22:19:40.375414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.375385 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:19:40.395921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.395881 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c5p75"] Oct 29 22:19:40.406223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.406196 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:19:40.410433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:40.410411 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-c6jrr"] Oct 29 22:19:42.387757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:42.387713 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699c48b1-116c-4902-9025-0505203814fd" path="/var/lib/kubelet/pods/699c48b1-116c-4902-9025-0505203814fd/volumes" Oct 29 22:19:42.388431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:42.388409 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" path="/var/lib/kubelet/pods/957524f0-a0ac-49fa-9dfd-50ff2fa43a52/volumes" Oct 29 22:19:43.195373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.195332 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl"] Oct 29 22:19:43.197542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.197514 4543 cpu_manager.go:414] "containerMap: removing container" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" Oct 29 22:19:43.197727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.197714 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" Oct 29 22:19:43.197851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.197840 4543 cpu_manager.go:414] "containerMap: removing container" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" Oct 29 22:19:43.197928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.197918 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" Oct 29 22:19:43.198042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.198032 4543 cpu_manager.go:414] "containerMap: removing container" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" Oct 29 22:19:43.198145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.198134 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" Oct 29 22:19:43.198736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.198716 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="957524f0-a0ac-49fa-9dfd-50ff2fa43a52" containerName="istio-proxy" Oct 29 22:19:43.198858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.198847 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="178fc4c0-85ee-4f87-932d-8a51e96cb0cf" containerName="istio-proxy" Oct 29 22:19:43.199020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.199007 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="699c48b1-116c-4902-9025-0505203814fd" containerName="istio-proxy" Oct 29 22:19:43.203889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.203871 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.204729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.204701 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl"] Oct 29 22:19:43.253286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.253258 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:19:43.253430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.253308 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:19:43.253430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.253351 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:19:43.253430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.253369 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:19:43.405056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.405016 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjr9g\" (UniqueName: \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.405519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.405249 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.405519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.405389 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.506932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.506899 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.507137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.507047 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjr9g\" (UniqueName: \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.507235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.507219 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.507877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.507841 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.509376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.509356 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.542302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.542267 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjr9g\" (UniqueName: \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\") pod \"maas-api-revoke-sub-6002b493-9bksl\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:43.820521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:43.820435 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:44.073995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.073912 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k"] Oct 29 22:19:44.078330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.078312 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.098786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.098757 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k"] Oct 29 22:19:44.117162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.117131 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.117312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.117181 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.117312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.117215 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r2n\" (UniqueName: \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.166137 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:19:44.166102 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc79acd_2e04_4adf_a239_4118d09d5c59.slice/crio-796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b WatchSource:0}: Error finding container 796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b: Status 404 returned error can't find the container with id 796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b Oct 29 22:19:44.166589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.166565 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl"] Oct 29 22:19:44.218796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.218767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.218897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.218821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.218897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.218870 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r2n\" (UniqueName: \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.219459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.219432 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.221279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.221262 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.323172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.323131 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" event={"ID":"2dc79acd-2e04-4adf-a239-4118d09d5c59","Type":"ContainerStarted","Data":"796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b"} Oct 29 22:19:44.353146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.352775 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r2n\" (UniqueName: \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\") pod \"maas-api-revoke-sub-516407b8-z4c7k\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.394325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.394296 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:44.761162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:44.761125 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k"] Oct 29 22:19:44.761596 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:19:44.761572 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a612e65_485f_41a4_a367_003f3e594478.slice/crio-1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9 WatchSource:0}: Error finding container 1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9: Status 404 returned error can't find the container with id 1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9 Oct 29 22:19:45.333831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:45.333796 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" event={"ID":"8a612e65-485f-41a4-a367-003f3e594478","Type":"ContainerStarted","Data":"dbc8c48f9fcdc1892c9d0583fcb1539898afa2e4aa734b4237163ce5de41baab"} Oct 29 22:19:45.333998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:45.333860 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" event={"ID":"8a612e65-485f-41a4-a367-003f3e594478","Type":"ContainerStarted","Data":"1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9"} Oct 29 22:19:45.336028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:45.335996 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" event={"ID":"2dc79acd-2e04-4adf-a239-4118d09d5c59","Type":"ContainerStarted","Data":"96b9b95da6ffa807a718d088f0bf4c404eed5ece5a94d8b181406549d13fdad0"} Oct 29 22:19:45.376005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:45.375950 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" podStartSLOduration=1.375929946 podStartE2EDuration="1.375929946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:45.374367149 +0000 UTC m=+2247.116297821" watchObservedRunningTime="2026-10-29 22:19:45.375929946 +0000 UTC m=+2247.117860620" Oct 29 22:19:45.416378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:45.416333 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" podStartSLOduration=2.416320037 podStartE2EDuration="2.416320037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:45.414845051 +0000 UTC m=+2247.156775736" watchObservedRunningTime="2026-10-29 22:19:45.416320037 +0000 UTC m=+2247.158250711" Oct 29 22:19:46.348251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:46.348218 4543 generic.go:356] "Generic (PLEG): container finished" podID="2dc79acd-2e04-4adf-a239-4118d09d5c59" containerID="96b9b95da6ffa807a718d088f0bf4c404eed5ece5a94d8b181406549d13fdad0" exitCode=0 Oct 29 22:19:46.348749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:46.348308 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" event={"ID":"2dc79acd-2e04-4adf-a239-4118d09d5c59","Type":"ContainerDied","Data":"96b9b95da6ffa807a718d088f0bf4c404eed5ece5a94d8b181406549d13fdad0"} Oct 29 22:19:46.350243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:46.350218 4543 generic.go:356] "Generic (PLEG): container finished" podID="8a612e65-485f-41a4-a367-003f3e594478" containerID="dbc8c48f9fcdc1892c9d0583fcb1539898afa2e4aa734b4237163ce5de41baab" exitCode=0 Oct 29 22:19:46.350411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:46.350297 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" event={"ID":"8a612e65-485f-41a4-a367-003f3e594478","Type":"ContainerDied","Data":"dbc8c48f9fcdc1892c9d0583fcb1539898afa2e4aa734b4237163ce5de41baab"} Oct 29 22:19:47.581474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.580599 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:47.637037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.636968 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:47.703066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.703037 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\") pod \"2dc79acd-2e04-4adf-a239-4118d09d5c59\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " Oct 29 22:19:47.703405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.703363 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\" (UniqueName: \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\") pod \"2dc79acd-2e04-4adf-a239-4118d09d5c59\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " Oct 29 22:19:47.703588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.703569 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\") pod \"2dc79acd-2e04-4adf-a239-4118d09d5c59\" (UID: \"2dc79acd-2e04-4adf-a239-4118d09d5c59\") " Oct 29 22:19:47.703696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.703406 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca" pod "2dc79acd-2e04-4adf-a239-4118d09d5c59" (UID: "2dc79acd-2e04-4adf-a239-4118d09d5c59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:47.703864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.703846 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\") pod \"8a612e65-485f-41a4-a367-003f3e594478\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " Oct 29 22:19:47.704373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.704352 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\" (UniqueName: \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\") pod \"8a612e65-485f-41a4-a367-003f3e594478\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " Oct 29 22:19:47.704486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.704410 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\") pod \"8a612e65-485f-41a4-a367-003f3e594478\" (UID: \"8a612e65-485f-41a4-a367-003f3e594478\") " Oct 29 22:19:47.704953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.704933 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca" pod "8a612e65-485f-41a4-a367-003f3e594478" (UID: "8a612e65-485f-41a4-a367-003f3e594478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:47.705368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.705349 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:47.705474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.705372 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a612e65-485f-41a4-a367-003f3e594478-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:47.707462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.707441 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca" pod "2dc79acd-2e04-4adf-a239-4118d09d5c59" (UID: "2dc79acd-2e04-4adf-a239-4118d09d5c59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:19:47.707594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.707576 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g" pod "2dc79acd-2e04-4adf-a239-4118d09d5c59" (UID: "2dc79acd-2e04-4adf-a239-4118d09d5c59"). InnerVolumeSpecName "kube-api-access-kjr9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:47.715715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.715672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca" pod "8a612e65-485f-41a4-a367-003f3e594478" (UID: "8a612e65-485f-41a4-a367-003f3e594478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:19:47.715833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.715812 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n" pod "8a612e65-485f-41a4-a367-003f3e594478" (UID: "8a612e65-485f-41a4-a367-003f3e594478"). InnerVolumeSpecName "kube-api-access-j9r2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:47.834102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.822477 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9r2n\" (UniqueName: \"kubernetes.io/projected/8a612e65-485f-41a4-a367-003f3e594478-kube-api-access-j9r2n\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:47.834102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.822516 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a612e65-485f-41a4-a367-003f3e594478-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:47.834102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.822530 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjr9g\" (UniqueName: \"kubernetes.io/projected/2dc79acd-2e04-4adf-a239-4118d09d5c59-kube-api-access-kjr9g\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:47.834102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:47.822544 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc79acd-2e04-4adf-a239-4118d09d5c59-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:48.120112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.115170 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:19:48.120112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.115251 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:19:48.376992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.376916 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" Oct 29 22:19:48.376992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.376947 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k" event={"ID":"8a612e65-485f-41a4-a367-003f3e594478","Type":"ContainerDied","Data":"1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9"} Oct 29 22:19:48.376992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.377004 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e6a355bbe890c092da8f8a317728378e8e0f5e688b277d393ae59607ba735a9" Oct 29 22:19:48.386395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.386370 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" Oct 29 22:19:48.396124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.396066 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl" event={"ID":"2dc79acd-2e04-4adf-a239-4118d09d5c59","Type":"ContainerDied","Data":"796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b"} Oct 29 22:19:48.396263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:48.396131 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796c7d4bfe123000b27661b04fe282215ee8ac0ece2b1203f0c9dfdfc54e5f1b" Oct 29 22:19:49.300861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:49.300825 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 22:19:49.301298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:49.301109 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" containerID="cri-o://c493474be9457120ebccea22ece55e80bad0a7cad7a11ac157167286b224f06b" gracePeriod=30 Oct 29 22:19:50.396547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.396484 4543 generic.go:356] "Generic (PLEG): container finished" podID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerID="15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a" exitCode=0 Oct 29 22:19:50.396946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.396555 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerDied","Data":"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a"} Oct 29 22:19:50.396946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.396882 4543 scope.go:122] "RemoveContainer" containerID="15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a" Oct 29 22:19:50.399043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.399005 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:50.399189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.399119 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:50.399189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:50.399173 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:19:51.426538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:51.426448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerStarted","Data":"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111"} Oct 29 22:19:52.382168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:52.382134 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:19:52.382382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:52.382202 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:19:52.387524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:52.387498 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:19:53.382344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.382304 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:19:53.382767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.382360 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:19:53.670156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.670062 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8"] Oct 29 22:19:53.670946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.670932 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2dc79acd-2e04-4adf-a239-4118d09d5c59" containerName="revoke-keys" Oct 29 22:19:53.670991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.670947 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc79acd-2e04-4adf-a239-4118d09d5c59" containerName="revoke-keys" Oct 29 22:19:53.670991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.670978 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8a612e65-485f-41a4-a367-003f3e594478" containerName="revoke-keys" Oct 29 22:19:53.670991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.670984 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a612e65-485f-41a4-a367-003f3e594478" containerName="revoke-keys" Oct 29 22:19:53.671147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.671137 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a612e65-485f-41a4-a367-003f3e594478" containerName="revoke-keys" Oct 29 22:19:53.671185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.671159 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc79acd-2e04-4adf-a239-4118d09d5c59" containerName="revoke-keys" Oct 29 22:19:53.674894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.674879 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.694423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.694398 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8"] Oct 29 22:19:53.695194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.695172 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4bz\" (UniqueName: \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.695250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.695219 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.695565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.695543 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.700677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.700656 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:19:53.700837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.700736 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:19:53.701695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.701677 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:19:53.701830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.701810 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:19:53.796823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.796787 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4bz\" (UniqueName: \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.796969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.796846 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.797025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.797011 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.797631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.797607 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.799673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.799639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.824207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.824186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4bz\" (UniqueName: \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\") pod \"maas-api-revoke-sub-f3336a63-n7cg8\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:53.988187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:53.988158 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:54.333672 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:19:54.333622 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527dbae9_e173_42df_8275_269f2e8b3f7f.slice/crio-f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327 WatchSource:0}: Error finding container f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327: Status 404 returned error can't find the container with id f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327 Oct 29 22:19:54.356540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.356401 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8"] Oct 29 22:19:54.358157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.358128 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:19:54.358285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.358178 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:19:54.381879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.381855 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:19:54.381978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.381907 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:19:54.405699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.405676 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng"] Oct 29 22:19:54.410898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.410878 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.451100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.429959 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng"] Oct 29 22:19:54.480774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.480746 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" event={"ID":"527dbae9-e173-42df-8275-269f2e8b3f7f","Type":"ContainerStarted","Data":"f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327"} Oct 29 22:19:54.483044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.482993 4543 generic.go:356] "Generic (PLEG): container finished" podID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerID="c493474be9457120ebccea22ece55e80bad0a7cad7a11ac157167286b224f06b" exitCode=0 Oct 29 22:19:54.483185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.483060 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerDied","Data":"c493474be9457120ebccea22ece55e80bad0a7cad7a11ac157167286b224f06b"} Oct 29 22:19:54.483185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.483144 4543 scope.go:122] "RemoveContainer" containerID="01c2ba99d4a3143c59091ce324cc15cef31fddb527dad5089af466911faa3236" Oct 29 22:19:54.522576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.522530 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.522815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.522685 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l9s\" (UniqueName: \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.566161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.564411 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.666147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.665888 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.666147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.665976 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l9s\" (UniqueName: \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.666147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.666126 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.666765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.666745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.669175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.669150 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.692763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.692745 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 22:19:54.693221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.693203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l9s\" (UniqueName: \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\") pod \"maas-api-revoke-sub-21754e44-jnpng\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.759647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.759329 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:54.766849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.766823 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.766983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767048 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767128 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767282 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767302 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:54.767395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767331 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767397 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767472 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767522 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\") pod \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\" (UID: \"97cc10bc-3596-4c2c-b4b9-4022dfed74fb\") " Oct 29 22:19:54.767610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.767538 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:54.768275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.768244 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.768275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.768268 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.768512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.768488 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:54.768703 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.768676 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:54.768950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.768932 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:54.771363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.771336 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:54.771467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.771390 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "kube-api-access-psczm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:54.771805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.771756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:54.772348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.772209 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token" pod "97cc10bc-3596-4c2c-b4b9-4022dfed74fb" (UID: "97cc10bc-3596-4c2c-b4b9-4022dfed74fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:54.879257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879212 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879247 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879262 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879277 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879293 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879306 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:54.879519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:54.879319 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psczm\" (UniqueName: \"kubernetes.io/projected/97cc10bc-3596-4c2c-b4b9-4022dfed74fb-kube-api-access-psczm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:55.101118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.101091 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng"] Oct 29 22:19:55.103339 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:19:55.103305 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79dd9f9_69fa_434d_9918_39f6b2bec85c.slice/crio-df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb WatchSource:0}: Error finding container df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb: Status 404 returned error can't find the container with id df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb Oct 29 22:19:55.381756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.381726 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:19:55.381901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.381775 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:19:55.494397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.494371 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" Oct 29 22:19:55.494803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.494409 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7" event={"ID":"97cc10bc-3596-4c2c-b4b9-4022dfed74fb","Type":"ContainerDied","Data":"2e00fc832cc3844ca543ea5a6f5428456c9380ef9ac6bfe1f87b0fde1b799a64"} Oct 29 22:19:55.494803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.494463 4543 scope.go:122] "RemoveContainer" containerID="c493474be9457120ebccea22ece55e80bad0a7cad7a11ac157167286b224f06b" Oct 29 22:19:55.496426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.496398 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" event={"ID":"527dbae9-e173-42df-8275-269f2e8b3f7f","Type":"ContainerStarted","Data":"27fc498ce37dfdaa2a21dc861e68661ee3c91698b5e5e72a6d3fa5ee3cb3dd4a"} Oct 29 22:19:55.498046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.498024 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" event={"ID":"a79dd9f9-69fa-434d-9918-39f6b2bec85c","Type":"ContainerStarted","Data":"df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb"} Oct 29 22:19:55.558225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.558119 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" podStartSLOduration=2.558103384 podStartE2EDuration="2.558103384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:55.550316505 +0000 UTC m=+2257.292247179" watchObservedRunningTime="2026-10-29 22:19:55.558103384 +0000 UTC m=+2257.300034059" Oct 29 22:19:55.591027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.590991 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 22:19:55.600408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:55.600379 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-bb2q7"] Oct 29 22:19:56.382148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.382123 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Oct 29 22:19:56.382278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.382172 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Oct 29 22:19:56.388286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.388263 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" path="/var/lib/kubelet/pods/97cc10bc-3596-4c2c-b4b9-4022dfed74fb/volumes" Oct 29 22:19:56.509520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.509488 4543 generic.go:356] "Generic (PLEG): container finished" podID="527dbae9-e173-42df-8275-269f2e8b3f7f" containerID="27fc498ce37dfdaa2a21dc861e68661ee3c91698b5e5e72a6d3fa5ee3cb3dd4a" exitCode=0 Oct 29 22:19:56.509988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.509526 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" event={"ID":"527dbae9-e173-42df-8275-269f2e8b3f7f","Type":"ContainerDied","Data":"27fc498ce37dfdaa2a21dc861e68661ee3c91698b5e5e72a6d3fa5ee3cb3dd4a"} Oct 29 22:19:56.511256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.511230 4543 generic.go:356] "Generic (PLEG): container finished" podID="a79dd9f9-69fa-434d-9918-39f6b2bec85c" containerID="8ee52fe0da518f1ae556f4b0ea2bb1fc62310483b085c942225ae9fb5774f307" exitCode=0 Oct 29 22:19:56.511386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:56.511294 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" event={"ID":"a79dd9f9-69fa-434d-9918-39f6b2bec85c","Type":"ContainerDied","Data":"8ee52fe0da518f1ae556f4b0ea2bb1fc62310483b085c942225ae9fb5774f307"} Oct 29 22:19:57.386702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.386670 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:19:57.387120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.387102 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:19:57.387802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.387781 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:19:57.787725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.787691 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:57.791646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.791627 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:57.936525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936448 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\") pod \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " Oct 29 22:19:57.936678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936527 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\") pod \"527dbae9-e173-42df-8275-269f2e8b3f7f\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " Oct 29 22:19:57.936678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936615 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\" (UniqueName: \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\") pod \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " Oct 29 22:19:57.936824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\") pod \"527dbae9-e173-42df-8275-269f2e8b3f7f\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " Oct 29 22:19:57.936931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936905 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca" pod "a79dd9f9-69fa-434d-9918-39f6b2bec85c" (UID: "a79dd9f9-69fa-434d-9918-39f6b2bec85c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:57.936985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.936953 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\") pod \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\" (UID: \"a79dd9f9-69fa-434d-9918-39f6b2bec85c\") " Oct 29 22:19:57.937308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.937245 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\" (UniqueName: \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\") pod \"527dbae9-e173-42df-8275-269f2e8b3f7f\" (UID: \"527dbae9-e173-42df-8275-269f2e8b3f7f\") " Oct 29 22:19:57.937308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.937286 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca" pod "527dbae9-e173-42df-8275-269f2e8b3f7f" (UID: "527dbae9-e173-42df-8275-269f2e8b3f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:57.938089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.938048 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:57.938193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.938099 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:57.939460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.939428 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca" pod "527dbae9-e173-42df-8275-269f2e8b3f7f" (UID: "527dbae9-e173-42df-8275-269f2e8b3f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:19:57.939606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.939588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s" pod "a79dd9f9-69fa-434d-9918-39f6b2bec85c" (UID: "a79dd9f9-69fa-434d-9918-39f6b2bec85c"). InnerVolumeSpecName "kube-api-access-v4l9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:57.939911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.939881 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz" pod "527dbae9-e173-42df-8275-269f2e8b3f7f" (UID: "527dbae9-e173-42df-8275-269f2e8b3f7f"). InnerVolumeSpecName "kube-api-access-tj4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:57.940540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:57.940524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca" pod "a79dd9f9-69fa-434d-9918-39f6b2bec85c" (UID: "a79dd9f9-69fa-434d-9918-39f6b2bec85c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:19:57.963988 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:19:57.963961 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:19:58.039779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.039753 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4bz\" (UniqueName: \"kubernetes.io/projected/527dbae9-e173-42df-8275-269f2e8b3f7f-kube-api-access-tj4bz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:58.039779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.039777 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527dbae9-e173-42df-8275-269f2e8b3f7f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:58.039928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.039788 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4l9s\" (UniqueName: \"kubernetes.io/projected/a79dd9f9-69fa-434d-9918-39f6b2bec85c-kube-api-access-v4l9s\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:58.039928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.039796 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79dd9f9-69fa-434d-9918-39f6b2bec85c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:19:58.538036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.538007 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" Oct 29 22:19:58.538185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.538043 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8" event={"ID":"527dbae9-e173-42df-8275-269f2e8b3f7f","Type":"ContainerDied","Data":"f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327"} Oct 29 22:19:58.538185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.538129 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f270e1d607bdacb82f370b22de3a934386cde35d63fc12783bd63032b9a85327" Oct 29 22:19:58.539642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.539626 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" Oct 29 22:19:58.539755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.539671 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng" event={"ID":"a79dd9f9-69fa-434d-9918-39f6b2bec85c","Type":"ContainerDied","Data":"df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb"} Oct 29 22:19:58.539755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:19:58.539725 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9a1dda101ccc8a0a0e0f384f312fc963df96e60060549ea89d89f1daf616bb" Oct 29 22:20:22.915694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:22.915660 4543 scope.go:122] "RemoveContainer" containerID="4c853e1edd7edc2235f33dcc484748456510aaa8036f190933a2420f4de5510e" Oct 29 22:20:22.930258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:22.930226 4543 scope.go:122] "RemoveContainer" containerID="56bf78a74874b42192d90cd3082ffca63bdf17712df92d80bd0bcc4e9f2fa0b5" Oct 29 22:20:22.940792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:22.940769 4543 scope.go:122] "RemoveContainer" containerID="9e4c1690601ad2a8ed32cd0d7d36fd7b02f8adb7290a91839634c3f2018127ad" Oct 29 22:20:27.962783 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:20:27.962745 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:20:44.163618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.163525 4543 generic.go:356] "Generic (PLEG): container finished" podID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerID="7be5bba1238928e4b663decfda0fb4698bae106748324f61f56936ef1e1ae11e" exitCode=0 Oct 29 22:20:44.163618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.163601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerDied","Data":"7be5bba1238928e4b663decfda0fb4698bae106748324f61f56936ef1e1ae11e"} Oct 29 22:20:44.164067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.163971 4543 scope.go:122] "RemoveContainer" containerID="7be5bba1238928e4b663decfda0fb4698bae106748324f61f56936ef1e1ae11e" Oct 29 22:20:44.165876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.165846 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:44.165955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.165922 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:44.165993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:44.165958 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:20:45.174001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:45.173967 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerStarted","Data":"d8b58decc33b5ad48745396e0bd4189f1dcbbdfb2e801e5b7826c483bd82f368"} Oct 29 22:20:45.353263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:45.353226 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:20:45.354199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:45.354167 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:45.354349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:45.354227 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:46.353027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:46.352420 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:46.353027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:46.352512 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:47.354539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:47.354502 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:47.355042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:47.354564 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:47.409225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:47.409198 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:20:48.354099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:48.353930 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:48.354099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:48.353984 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:49.350348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:49.350314 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:49.350926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:49.350896 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:50.350625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:50.350587 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" start-of-body= Oct 29 22:20:50.351015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:50.350651 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.62:15021/healthz/ready\": dial tcp 10.133.1.62:15021: connect: connection refused" Oct 29 22:20:51.355820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:51.355790 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:20:51.357263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:51.357232 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:20:56.302811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.302772 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:20:56.304973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.304943 4543 cpu_manager.go:414] "containerMap: removing container" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.305181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305163 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.305332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305316 4543 cpu_manager.go:414] "containerMap: removing container" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.305439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305428 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.305547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305538 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a79dd9f9-69fa-434d-9918-39f6b2bec85c" containerName="revoke-keys" Oct 29 22:20:56.305626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305617 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79dd9f9-69fa-434d-9918-39f6b2bec85c" containerName="revoke-keys" Oct 29 22:20:56.305730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305721 4543 cpu_manager.go:414] "containerMap: removing container" podUID="527dbae9-e173-42df-8275-269f2e8b3f7f" containerName="revoke-keys" Oct 29 22:20:56.305808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.305798 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527dbae9-e173-42df-8275-269f2e8b3f7f" containerName="revoke-keys" Oct 29 22:20:56.306180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.306163 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.306329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.306317 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79dd9f9-69fa-434d-9918-39f6b2bec85c" containerName="revoke-keys" Oct 29 22:20:56.306434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.306424 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="527dbae9-e173-42df-8275-269f2e8b3f7f" containerName="revoke-keys" Oct 29 22:20:56.310047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.310029 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.322642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.322622 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:20:56.328526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.328503 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b9ed94\"" Oct 29 22:20:56.328623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.328542 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b9ed94-dockercfg-h6hgm\"" Oct 29 22:20:56.389944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.389912 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:20:56.391382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.391354 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cc10bc-3596-4c2c-b4b9-4022dfed74fb" containerName="istio-proxy" Oct 29 22:20:56.395159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.395134 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.499100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.463048 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:20:56.499100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.481906 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4v9n\" (UniqueName: \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.499100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.482109 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.499100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.482244 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.499100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.482268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vsn\" (UniqueName: \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.583789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.583426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.584250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.584215 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.586648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.585207 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vsn\" (UniqueName: \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.586648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.585351 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4v9n\" (UniqueName: \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.586648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.585174 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.587578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.587534 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.637276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.637238 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vsn\" (UniqueName: \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\") pod \"payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:56.641280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.641197 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4v9n\" (UniqueName: \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\") pod \"payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.837524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.837457 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:56.925831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:56.925755 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:57.165514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.165483 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:20:57.165909 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:20:57.165884 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63633283_238a_4630_975a_b9bb3c43d9b9.slice/crio-ef12e84ba343cd18acc773569beffa06438aad3a268161eee24b57a0ff540ac7 WatchSource:0}: Error finding container ef12e84ba343cd18acc773569beffa06438aad3a268161eee24b57a0ff540ac7: Status 404 returned error can't find the container with id ef12e84ba343cd18acc773569beffa06438aad3a268161eee24b57a0ff540ac7 Oct 29 22:20:57.258877 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:20:57.258853 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e8126b_2602_4a76_af63_860ae670c6d3.slice/crio-1d4fd7711a3d6f05a0664e1d42c217b9aebd13d0a477b4a16042eb10c4750605 WatchSource:0}: Error finding container 1d4fd7711a3d6f05a0664e1d42c217b9aebd13d0a477b4a16042eb10c4750605: Status 404 returned error can't find the container with id 1d4fd7711a3d6f05a0664e1d42c217b9aebd13d0a477b4a16042eb10c4750605 Oct 29 22:20:57.259285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.259264 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:20:57.400781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.400745 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" event={"ID":"63633283-238a-4630-975a-b9bb3c43d9b9","Type":"ContainerStarted","Data":"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af"} Oct 29 22:20:57.401449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.401425 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" event={"ID":"63633283-238a-4630-975a-b9bb3c43d9b9","Type":"ContainerStarted","Data":"ef12e84ba343cd18acc773569beffa06438aad3a268161eee24b57a0ff540ac7"} Oct 29 22:20:57.402175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.402136 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:20:57.429057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.428953 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" event={"ID":"18e8126b-2602-4a76-af63-860ae670c6d3","Type":"ContainerStarted","Data":"1d4fd7711a3d6f05a0664e1d42c217b9aebd13d0a477b4a16042eb10c4750605"} Oct 29 22:20:57.484163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.484099 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:20:57.490321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.487635 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" podStartSLOduration=1.4876182660000001 podStartE2EDuration="1.487618266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:57.467453432 +0000 UTC m=+2319.209384105" watchObservedRunningTime="2026-10-29 22:20:57.487618266 +0000 UTC m=+2319.229548941" Oct 29 22:20:57.496670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.495735 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.496670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.496150 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:20:57.512346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.511954 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-e2e--d445abcd-dockercfg-f5q7z\"" Oct 29 22:20:57.512346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.511984 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-58ea3cdf2b14\"" Oct 29 22:20:57.512491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.512460 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"openshift-service-ca.crt\"" Oct 29 22:20:57.512588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.512573 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"routing-overlay\"" Oct 29 22:20:57.512681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.512667 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-plugins-e2e-extmod-b-07e166\"" Oct 29 22:20:57.512775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.512761 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"kube-root-ca.crt\"" Oct 29 22:20:57.628101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.627616 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.628101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.627700 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.628101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.627790 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvlc\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.628101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.627942 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.734099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.733332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.734099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.733607 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.734099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.733702 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.734099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.733810 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvlc\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.737103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.735001 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.737103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.736299 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.759007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.758905 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.779054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.779022 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvlc\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.816098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:57.815068 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:57.964505 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:20:57.964473 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:20:58.136986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.136958 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:20:58.137443 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:20:58.137419 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4f73c4_ed60_4c5a_865c_44ca60f54603.slice/crio-8284639e920c2b26830fced9564f912d8e93532a258852b6c11a948b73db7cd1 WatchSource:0}: Error finding container 8284639e920c2b26830fced9564f912d8e93532a258852b6c11a948b73db7cd1: Status 404 returned error can't find the container with id 8284639e920c2b26830fced9564f912d8e93532a258852b6c11a948b73db7cd1 Oct 29 22:20:58.438008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.437923 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" event={"ID":"0c4f73c4-ed60-4c5a-865c-44ca60f54603","Type":"ContainerStarted","Data":"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734"} Oct 29 22:20:58.438008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.437985 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" event={"ID":"0c4f73c4-ed60-4c5a-865c-44ca60f54603","Type":"ContainerStarted","Data":"8284639e920c2b26830fced9564f912d8e93532a258852b6c11a948b73db7cd1"} Oct 29 22:20:58.438505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.438115 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:20:58.439629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.439607 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" event={"ID":"18e8126b-2602-4a76-af63-860ae670c6d3","Type":"ContainerStarted","Data":"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95"} Oct 29 22:20:58.439733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.439647 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:20:58.480959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.480917 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" podStartSLOduration=1.480905209 podStartE2EDuration="1.480905209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:58.478907915 +0000 UTC m=+2320.220838587" watchObservedRunningTime="2026-10-29 22:20:58.480905209 +0000 UTC m=+2320.222835905" Oct 29 22:20:58.519949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:20:58.519881 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" podStartSLOduration=2.519863361 podStartE2EDuration="2.519863361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:58.518860002 +0000 UTC m=+2320.260790687" watchObservedRunningTime="2026-10-29 22:20:58.519863361 +0000 UTC m=+2320.261794036" Oct 29 22:21:09.452775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:09.452744 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:21:09.453224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:09.452826 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:21:09.453224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:09.452970 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:21:22.221465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:22.221429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 22:21:22.222330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:22.222273 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" containerID="cri-o://9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016" gracePeriod=30 Oct 29 22:21:27.515246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.515219 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 22:21:27.552165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552139 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552178 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552224 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552329 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552497 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552497 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:27.552579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552543 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552530 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:27.552579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552563 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:21:27.552768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552602 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552752 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:27.552913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552804 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.552913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552880 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:27.552913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.552897 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\") pod \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\" (UID: \"3b837bfb-9a20-436a-b39e-85c1e3daa3e7\") " Oct 29 22:21:27.553575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.553548 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.553713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.553594 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.553713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.553670 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.553867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.553761 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.553867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.553776 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.554859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.554835 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "kube-api-access-7x8rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:27.555001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.554981 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:21:27.555047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.555024 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:27.555317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.555304 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy" pod "3b837bfb-9a20-436a-b39e-85c1e3daa3e7" (UID: "3b837bfb-9a20-436a-b39e-85c1e3daa3e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:27.655004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.654974 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.655004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.655001 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8rg\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-kube-api-access-7x8rg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.655211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.655012 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.655211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.655040 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b837bfb-9a20-436a-b39e-85c1e3daa3e7-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:27.713520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.713484 4543 generic.go:356] "Generic (PLEG): container finished" podID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerID="9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016" exitCode=0 Oct 29 22:21:27.713721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.713548 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" Oct 29 22:21:27.713721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.713565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerDied","Data":"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016"} Oct 29 22:21:27.713721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.713649 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2" event={"ID":"3b837bfb-9a20-436a-b39e-85c1e3daa3e7","Type":"ContainerDied","Data":"1fd02083f1f93c263d3155a18bcc39553de927a7e15a7e517095fa58a51d71b7"} Oct 29 22:21:27.713721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.713673 4543 scope.go:122] "RemoveContainer" containerID="9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016" Oct 29 22:21:27.726125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.726107 4543 scope.go:122] "RemoveContainer" containerID="864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c" Oct 29 22:21:27.736295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.736272 4543 scope.go:122] "RemoveContainer" containerID="9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016" Oct 29 22:21:27.736565 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:21:27.736543 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016\": container with ID starting with 9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016 not found: ID does not exist" containerID="9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016" Oct 29 22:21:27.736614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.736573 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016"} err="failed to get container status \"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016\": rpc error: code = NotFound desc = could not find container \"9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016\": container with ID starting with 9f77c874593bfb11311aee0cc38ddbd88c254268cefd71d02637d1dca8fa5016 not found: ID does not exist" Oct 29 22:21:27.736614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.736590 4543 scope.go:122] "RemoveContainer" containerID="864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c" Oct 29 22:21:27.736850 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:21:27.736834 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c\": container with ID starting with 864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c not found: ID does not exist" containerID="864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c" Oct 29 22:21:27.736896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.736854 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c"} err="failed to get container status \"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c\": rpc error: code = NotFound desc = could not find container \"864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c\": container with ID starting with 864315b3036e5509554f3c6ee2ed78e7f623da8e660efd507a855921afd40a6c not found: ID does not exist" Oct 29 22:21:27.774829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.774782 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 22:21:27.784324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:27.784301 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-sslm2"] Oct 29 22:21:27.963062 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:21:27.963031 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:21:28.390772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:28.390742 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" path="/var/lib/kubelet/pods/3b837bfb-9a20-436a-b39e-85c1e3daa3e7/volumes" Oct 29 22:21:40.118494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.118460 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt"] Oct 29 22:21:40.119393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119376 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.119393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119393 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.119514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119404 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.119514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119409 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.119581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119521 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.119581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.119532 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b837bfb-9a20-436a-b39e-85c1e3daa3e7" containerName="istio-proxy" Oct 29 22:21:40.125220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.125201 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.142347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.142325 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt"] Oct 29 22:21:40.148751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.148726 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:21:40.148876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.148748 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:21:40.148876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.148780 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:21:40.152148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.152127 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:21:40.270812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.270786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.270929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.270820 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8q7\" (UniqueName: \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.270977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.270948 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.372485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.372421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.372485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.372458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8q7\" (UniqueName: \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.372678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.372523 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.373247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.373170 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.374888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.374869 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.396877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.396853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8q7\" (UniqueName: \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\") pod \"maas-api-revoke-sub-67b8fea0-9w9wt\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.441476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.441452 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:40.762864 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:21:40.760647 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e2bab6_4000_412d_8215_331ecb6d99fa.slice/crio-38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71 WatchSource:0}: Error finding container 38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71: Status 404 returned error can't find the container with id 38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71 Oct 29 22:21:40.762864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.762828 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt"] Oct 29 22:21:40.864057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:40.864034 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" event={"ID":"20e2bab6-4000-412d-8215-331ecb6d99fa","Type":"ContainerStarted","Data":"38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71"} Oct 29 22:21:41.874485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:41.874448 4543 generic.go:356] "Generic (PLEG): container finished" podID="20e2bab6-4000-412d-8215-331ecb6d99fa" containerID="cf6dbdc2ec0c73c241087899353d8a989c99430dafc545892764575c670ab94e" exitCode=0 Oct 29 22:21:41.874927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:41.874525 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" event={"ID":"20e2bab6-4000-412d-8215-331ecb6d99fa","Type":"ContainerDied","Data":"cf6dbdc2ec0c73c241087899353d8a989c99430dafc545892764575c670ab94e"} Oct 29 22:21:43.105459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.105436 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:43.125440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.125413 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\") pod \"20e2bab6-4000-412d-8215-331ecb6d99fa\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " Oct 29 22:21:43.125585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.125479 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\" (UniqueName: \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\") pod \"20e2bab6-4000-412d-8215-331ecb6d99fa\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " Oct 29 22:21:43.125662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.125600 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\") pod \"20e2bab6-4000-412d-8215-331ecb6d99fa\" (UID: \"20e2bab6-4000-412d-8215-331ecb6d99fa\") " Oct 29 22:21:43.125785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.125761 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca" pod "20e2bab6-4000-412d-8215-331ecb6d99fa" (UID: "20e2bab6-4000-412d-8215-331ecb6d99fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:21:43.126207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.126189 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:43.127634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.127613 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca" pod "20e2bab6-4000-412d-8215-331ecb6d99fa" (UID: "20e2bab6-4000-412d-8215-331ecb6d99fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:43.127780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.127761 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7" pod "20e2bab6-4000-412d-8215-331ecb6d99fa" (UID: "20e2bab6-4000-412d-8215-331ecb6d99fa"). InnerVolumeSpecName "kube-api-access-zw8q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:43.227630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.227582 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e2bab6-4000-412d-8215-331ecb6d99fa-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:43.227630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.227612 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8q7\" (UniqueName: \"kubernetes.io/projected/20e2bab6-4000-412d-8215-331ecb6d99fa-kube-api-access-zw8q7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:43.895091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.895054 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" Oct 29 22:21:43.895272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.895099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt" event={"ID":"20e2bab6-4000-412d-8215-331ecb6d99fa","Type":"ContainerDied","Data":"38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71"} Oct 29 22:21:43.895272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:43.895152 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38aad1a369107ca8614f4069dd2d7a4c593651cf03774c936b2441af33e5cc71" Oct 29 22:21:50.447464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.447433 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv"] Oct 29 22:21:50.448339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.448322 4543 cpu_manager.go:414] "containerMap: removing container" podUID="20e2bab6-4000-412d-8215-331ecb6d99fa" containerName="revoke-keys" Oct 29 22:21:50.448339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.448340 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e2bab6-4000-412d-8215-331ecb6d99fa" containerName="revoke-keys" Oct 29 22:21:50.448465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.448458 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e2bab6-4000-412d-8215-331ecb6d99fa" containerName="revoke-keys" Oct 29 22:21:50.452166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.452150 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.471423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.471395 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv"] Oct 29 22:21:50.477769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.477749 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:21:50.477769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.477761 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:21:50.477941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.477758 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:21:50.479458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.479436 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:21:50.527638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.527617 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.527819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.527800 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.527867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.527853 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78z6\" (UniqueName: \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.628992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.628952 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.629199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.629179 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.629265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.629247 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78z6\" (UniqueName: \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.629764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.629745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.631420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.631402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.653507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.653483 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78z6\" (UniqueName: \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\") pod \"maas-api-revoke-sub-ed43f451-md8fv\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:50.767947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:50.767927 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:51.093418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:51.093361 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv"] Oct 29 22:21:51.093616 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:21:51.093595 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee42192e_59ef_41ba_b917_268e18a8c2fa.slice/crio-acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2 WatchSource:0}: Error finding container acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2: Status 404 returned error can't find the container with id acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2 Oct 29 22:21:51.980041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:51.980014 4543 generic.go:356] "Generic (PLEG): container finished" podID="ee42192e-59ef-41ba-b917-268e18a8c2fa" containerID="39c3fd1584fd4dd988b68a3f4863637009796f60168ed6110c28eb3a7add50a3" exitCode=0 Oct 29 22:21:51.980463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:51.980118 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" event={"ID":"ee42192e-59ef-41ba-b917-268e18a8c2fa","Type":"ContainerDied","Data":"39c3fd1584fd4dd988b68a3f4863637009796f60168ed6110c28eb3a7add50a3"} Oct 29 22:21:51.980463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:51.980165 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" event={"ID":"ee42192e-59ef-41ba-b917-268e18a8c2fa","Type":"ContainerStarted","Data":"acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2"} Oct 29 22:21:53.173187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.173161 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:53.265035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.264961 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\") pod \"ee42192e-59ef-41ba-b917-268e18a8c2fa\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " Oct 29 22:21:53.265035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.265008 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\" (UniqueName: \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\") pod \"ee42192e-59ef-41ba-b917-268e18a8c2fa\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " Oct 29 22:21:53.265242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.265051 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\") pod \"ee42192e-59ef-41ba-b917-268e18a8c2fa\" (UID: \"ee42192e-59ef-41ba-b917-268e18a8c2fa\") " Oct 29 22:21:53.265746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.265714 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca" pod "ee42192e-59ef-41ba-b917-268e18a8c2fa" (UID: "ee42192e-59ef-41ba-b917-268e18a8c2fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:21:53.266010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.265986 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:53.267430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.267407 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca" pod "ee42192e-59ef-41ba-b917-268e18a8c2fa" (UID: "ee42192e-59ef-41ba-b917-268e18a8c2fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:53.267549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.267493 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6" pod "ee42192e-59ef-41ba-b917-268e18a8c2fa" (UID: "ee42192e-59ef-41ba-b917-268e18a8c2fa"). InnerVolumeSpecName "kube-api-access-k78z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:53.367575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.367546 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee42192e-59ef-41ba-b917-268e18a8c2fa-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:53.367575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.367572 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78z6\" (UniqueName: \"kubernetes.io/projected/ee42192e-59ef-41ba-b917-268e18a8c2fa-kube-api-access-k78z6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:21:53.998600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.998571 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" Oct 29 22:21:53.998600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.998584 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv" event={"ID":"ee42192e-59ef-41ba-b917-268e18a8c2fa","Type":"ContainerDied","Data":"acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2"} Oct 29 22:21:53.998824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:21:53.998643 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc7d9ac2bcbe241275c0a1123592daafd98ebadcbcfeb7999fce0a6bd846cd2" Oct 29 22:21:57.961576 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:21:57.961540 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:22:09.163927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.163850 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f"] Oct 29 22:22:09.164779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.164762 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ee42192e-59ef-41ba-b917-268e18a8c2fa" containerName="revoke-keys" Oct 29 22:22:09.164842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.164781 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee42192e-59ef-41ba-b917-268e18a8c2fa" containerName="revoke-keys" Oct 29 22:22:09.164925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.164915 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee42192e-59ef-41ba-b917-268e18a8c2fa" containerName="revoke-keys" Oct 29 22:22:09.168589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.168573 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.189152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.189126 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f"] Oct 29 22:22:09.193896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.193875 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:22:09.194065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.193856 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:22:09.194065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.193935 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:22:09.195974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.195955 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:22:09.249849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.249821 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.249955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.249899 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgqb\" (UniqueName: \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.250141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.250126 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.351375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.351339 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.351559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.351525 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.351604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.351577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgqb\" (UniqueName: \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.352286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.352257 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.354280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.354255 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.377303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.377268 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgqb\" (UniqueName: \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\") pod \"maas-api-revoke-sub-39a3d33e-zg74f\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.490840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.490814 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:09.805725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:09.805694 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f"] Oct 29 22:22:09.806476 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:22:09.806453 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98443e6c_f276_43a7_b60d_eaff0bc36537.slice/crio-a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2 WatchSource:0}: Error finding container a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2: Status 404 returned error can't find the container with id a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2 Oct 29 22:22:10.273566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.273532 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" event={"ID":"98443e6c-f276-43a7-b60d-eaff0bc36537","Type":"ContainerStarted","Data":"b222f11c86cdfda4019ecd901e3aedf5a0195725ce5d57018dc33805b5ec28e3"} Oct 29 22:22:10.273918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.273596 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" event={"ID":"98443e6c-f276-43a7-b60d-eaff0bc36537","Type":"ContainerStarted","Data":"a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2"} Oct 29 22:22:10.317934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.317870 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" podStartSLOduration=1.317855046 podStartE2EDuration="1.317855046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:10.313511987 +0000 UTC m=+2392.055442660" watchObservedRunningTime="2026-10-29 22:22:10.317855046 +0000 UTC m=+2392.059785748" Oct 29 22:22:10.621350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.621324 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk"] Oct 29 22:22:10.626451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.626432 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.649124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.649094 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk"] Oct 29 22:22:10.771651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.771621 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xt7\" (UniqueName: \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.771792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.771671 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.771792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.771747 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.873383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.873311 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xt7\" (UniqueName: \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.873383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.873359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.873549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.873388 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.874188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.874163 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.876097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.876059 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.899531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.899508 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xt7\" (UniqueName: \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\") pod \"maas-api-revoke-sub-19b8305d-6hmjk\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:10.958371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:10.958349 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:11.283380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:11.283349 4543 generic.go:356] "Generic (PLEG): container finished" podID="98443e6c-f276-43a7-b60d-eaff0bc36537" containerID="b222f11c86cdfda4019ecd901e3aedf5a0195725ce5d57018dc33805b5ec28e3" exitCode=0 Oct 29 22:22:11.283738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:11.283433 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" event={"ID":"98443e6c-f276-43a7-b60d-eaff0bc36537","Type":"ContainerDied","Data":"b222f11c86cdfda4019ecd901e3aedf5a0195725ce5d57018dc33805b5ec28e3"} Oct 29 22:22:11.345695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:11.345669 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk"] Oct 29 22:22:11.346465 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:22:11.346441 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c000437_c23b_4f26_9ca0_2307174db1ff.slice/crio-ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530 WatchSource:0}: Error finding container ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530: Status 404 returned error can't find the container with id ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530 Oct 29 22:22:12.294022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.293984 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" event={"ID":"6c000437-c23b-4f26-9ca0-2307174db1ff","Type":"ContainerStarted","Data":"f8816485831c8dce0cecb6f5318d11b99b89e420c5e73da05b52f238bb50b69b"} Oct 29 22:22:12.294650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.294050 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" event={"ID":"6c000437-c23b-4f26-9ca0-2307174db1ff","Type":"ContainerStarted","Data":"ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530"} Oct 29 22:22:12.337065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.337015 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" podStartSLOduration=2.336997642 podStartE2EDuration="2.336997642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:12.334453353 +0000 UTC m=+2394.076384023" watchObservedRunningTime="2026-10-29 22:22:12.336997642 +0000 UTC m=+2394.078928315" Oct 29 22:22:12.468614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.468589 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:12.601996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.601924 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\" (UniqueName: \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\") pod \"98443e6c-f276-43a7-b60d-eaff0bc36537\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " Oct 29 22:22:12.602193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.602177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\") pod \"98443e6c-f276-43a7-b60d-eaff0bc36537\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " Oct 29 22:22:12.602278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.602261 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\") pod \"98443e6c-f276-43a7-b60d-eaff0bc36537\" (UID: \"98443e6c-f276-43a7-b60d-eaff0bc36537\") " Oct 29 22:22:12.602630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.602602 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca" pod "98443e6c-f276-43a7-b60d-eaff0bc36537" (UID: "98443e6c-f276-43a7-b60d-eaff0bc36537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:12.602930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.602708 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:12.604251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.604228 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb" pod "98443e6c-f276-43a7-b60d-eaff0bc36537" (UID: "98443e6c-f276-43a7-b60d-eaff0bc36537"). InnerVolumeSpecName "kube-api-access-tqgqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:12.604350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.604284 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca" pod "98443e6c-f276-43a7-b60d-eaff0bc36537" (UID: "98443e6c-f276-43a7-b60d-eaff0bc36537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:22:12.704573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.704530 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98443e6c-f276-43a7-b60d-eaff0bc36537-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:12.704573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:12.704568 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqgqb\" (UniqueName: \"kubernetes.io/projected/98443e6c-f276-43a7-b60d-eaff0bc36537-kube-api-access-tqgqb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:13.304257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:13.304215 4543 generic.go:356] "Generic (PLEG): container finished" podID="6c000437-c23b-4f26-9ca0-2307174db1ff" containerID="f8816485831c8dce0cecb6f5318d11b99b89e420c5e73da05b52f238bb50b69b" exitCode=0 Oct 29 22:22:13.304710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:13.304360 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" event={"ID":"6c000437-c23b-4f26-9ca0-2307174db1ff","Type":"ContainerDied","Data":"f8816485831c8dce0cecb6f5318d11b99b89e420c5e73da05b52f238bb50b69b"} Oct 29 22:22:13.306228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:13.306209 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" Oct 29 22:22:13.306384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:13.306264 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f" event={"ID":"98443e6c-f276-43a7-b60d-eaff0bc36537","Type":"ContainerDied","Data":"a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2"} Oct 29 22:22:13.306384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:13.306310 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a471bdf7c1dc00f2948df098e34dd133b922909c1903af92defa53542e87aca2" Oct 29 22:22:14.574425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.574352 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:14.767402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.767365 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\") pod \"6c000437-c23b-4f26-9ca0-2307174db1ff\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " Oct 29 22:22:14.767663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.767645 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\" (UniqueName: \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\") pod \"6c000437-c23b-4f26-9ca0-2307174db1ff\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " Oct 29 22:22:14.767748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.767685 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\") pod \"6c000437-c23b-4f26-9ca0-2307174db1ff\" (UID: \"6c000437-c23b-4f26-9ca0-2307174db1ff\") " Oct 29 22:22:14.767811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.767775 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca" pod "6c000437-c23b-4f26-9ca0-2307174db1ff" (UID: "6c000437-c23b-4f26-9ca0-2307174db1ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:14.768361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.768328 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.770355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.770331 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca" pod "6c000437-c23b-4f26-9ca0-2307174db1ff" (UID: "6c000437-c23b-4f26-9ca0-2307174db1ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:22:14.770519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.770502 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7" pod "6c000437-c23b-4f26-9ca0-2307174db1ff" (UID: "6c000437-c23b-4f26-9ca0-2307174db1ff"). InnerVolumeSpecName "kube-api-access-q8xt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:14.870621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.870530 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xt7\" (UniqueName: \"kubernetes.io/projected/6c000437-c23b-4f26-9ca0-2307174db1ff-kube-api-access-q8xt7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.870621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:14.870577 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c000437-c23b-4f26-9ca0-2307174db1ff-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:15.325131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:15.325100 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" event={"ID":"6c000437-c23b-4f26-9ca0-2307174db1ff","Type":"ContainerDied","Data":"ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530"} Oct 29 22:22:15.325131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:15.325118 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk" Oct 29 22:22:15.325347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:15.325170 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab55d7b801b28c076ad7d17045f7c72b7e4cec0208caef2b57c6b34f22cd9530" Oct 29 22:22:21.162672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.162224 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl"] Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.163827 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6c000437-c23b-4f26-9ca0-2307174db1ff" containerName="revoke-keys" Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.163860 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c000437-c23b-4f26-9ca0-2307174db1ff" containerName="revoke-keys" Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.163964 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98443e6c-f276-43a7-b60d-eaff0bc36537" containerName="revoke-keys" Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.163974 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98443e6c-f276-43a7-b60d-eaff0bc36537" containerName="revoke-keys" Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.164251 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98443e6c-f276-43a7-b60d-eaff0bc36537" containerName="revoke-keys" Oct 29 22:22:21.164666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.164281 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c000437-c23b-4f26-9ca0-2307174db1ff" containerName="revoke-keys" Oct 29 22:22:21.196175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.195887 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl"] Oct 29 22:22:21.196175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.196049 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.211947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.211923 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:22:21.223840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.223815 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:22:21.224018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.223996 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:22:21.224146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.224102 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:22:21.230188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.230154 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdzb\" (UniqueName: \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.230376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.230354 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.230535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.230519 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.331520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.331486 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.331679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.331662 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdzb\" (UniqueName: \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.331773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.331759 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.332109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.332070 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.334387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.334360 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.358623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.358602 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdzb\" (UniqueName: \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\") pod \"maas-api-revoke-sub-38504de1-7xccl\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.526573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.517956 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:21.903517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:21.903488 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl"] Oct 29 22:22:21.904565 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:22:21.904540 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3872383_adf2_4a9a_8dab_b7a685ae60ee.slice/crio-2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca WatchSource:0}: Error finding container 2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca: Status 404 returned error can't find the container with id 2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca Oct 29 22:22:22.545614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:22.545539 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" event={"ID":"d3872383-adf2-4a9a-8dab-b7a685ae60ee","Type":"ContainerStarted","Data":"c7515cb966fd77f0085576f14cd8f8d8ccdfd27cf6b488cf187f4a7024db2e30"} Oct 29 22:22:22.545614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:22.545598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" event={"ID":"d3872383-adf2-4a9a-8dab-b7a685ae60ee","Type":"ContainerStarted","Data":"2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca"} Oct 29 22:22:22.588961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:22.588917 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" podStartSLOduration=1.588902335 podStartE2EDuration="1.588902335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:22.585849762 +0000 UTC m=+2404.327780435" watchObservedRunningTime="2026-10-29 22:22:22.588902335 +0000 UTC m=+2404.330833049" Oct 29 22:22:23.018640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:23.018604 4543 scope.go:122] "RemoveContainer" containerID="ceb69a7f06b144a9a60f20c7a72ead1c349cf289cbdcbcac8ad0e49b549c573f" Oct 29 22:22:23.556383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:23.556356 4543 generic.go:356] "Generic (PLEG): container finished" podID="d3872383-adf2-4a9a-8dab-b7a685ae60ee" containerID="c7515cb966fd77f0085576f14cd8f8d8ccdfd27cf6b488cf187f4a7024db2e30" exitCode=0 Oct 29 22:22:23.556754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:23.556444 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" event={"ID":"d3872383-adf2-4a9a-8dab-b7a685ae60ee","Type":"ContainerDied","Data":"c7515cb966fd77f0085576f14cd8f8d8ccdfd27cf6b488cf187f4a7024db2e30"} Oct 29 22:22:24.804670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.804639 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:24.912188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.912100 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\") pod \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " Oct 29 22:22:24.912355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.912209 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\") pod \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " Oct 29 22:22:24.912355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.912337 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\" (UniqueName: \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\") pod \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\" (UID: \"d3872383-adf2-4a9a-8dab-b7a685ae60ee\") " Oct 29 22:22:24.912516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.912493 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca" pod "d3872383-adf2-4a9a-8dab-b7a685ae60ee" (UID: "d3872383-adf2-4a9a-8dab-b7a685ae60ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:24.913066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.913036 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:24.915101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.915062 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca" pod "d3872383-adf2-4a9a-8dab-b7a685ae60ee" (UID: "d3872383-adf2-4a9a-8dab-b7a685ae60ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:22:24.915199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:24.915159 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb" pod "d3872383-adf2-4a9a-8dab-b7a685ae60ee" (UID: "d3872383-adf2-4a9a-8dab-b7a685ae60ee"). InnerVolumeSpecName "kube-api-access-4rdzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:25.048099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:25.043548 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3872383-adf2-4a9a-8dab-b7a685ae60ee-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.048099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:25.043582 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rdzb\" (UniqueName: \"kubernetes.io/projected/d3872383-adf2-4a9a-8dab-b7a685ae60ee-kube-api-access-4rdzb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.586312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:25.586252 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" Oct 29 22:22:25.586312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:25.586282 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl" event={"ID":"d3872383-adf2-4a9a-8dab-b7a685ae60ee","Type":"ContainerDied","Data":"2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca"} Oct 29 22:22:25.586614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:25.586345 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea4ac9cb9209597c6e86cf9936f1e84548ed49b95af5ef6cbac1cd3f619cfca" Oct 29 22:22:27.068497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.068461 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:22:27.069895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.069870 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d3872383-adf2-4a9a-8dab-b7a685ae60ee" containerName="revoke-keys" Oct 29 22:22:27.069998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.069899 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3872383-adf2-4a9a-8dab-b7a685ae60ee" containerName="revoke-keys" Oct 29 22:22:27.070151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.070137 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3872383-adf2-4a9a-8dab-b7a685ae60ee" containerName="revoke-keys" Oct 29 22:22:27.074980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.074964 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.091669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.091643 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:22:27.097200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.097177 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-50d4af\"" Oct 29 22:22:27.195645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.195409 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-50d4af-dockercfg-qz99g\"" Oct 29 22:22:27.253104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.245499 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xb4z\" (UniqueName: \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.253104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.245637 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.382196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.382061 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xb4z\" (UniqueName: \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.382551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.382529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.384130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.384106 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.407995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.407960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xb4z\" (UniqueName: \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\") pod \"payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.690843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:27.690761 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:27.963712 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:22:27.963677 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:22:28.006696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:28.006672 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:22:28.007229 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:22:28.007205 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ca6692_75b0_425e_82ec_fbab68fbe5a5.slice/crio-7cd247de83012d5806799bc7148f30cb1d73065d960cf6731c6c3645db0019fa WatchSource:0}: Error finding container 7cd247de83012d5806799bc7148f30cb1d73065d960cf6731c6c3645db0019fa: Status 404 returned error can't find the container with id 7cd247de83012d5806799bc7148f30cb1d73065d960cf6731c6c3645db0019fa Oct 29 22:22:28.647980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:28.647946 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" event={"ID":"a9ca6692-75b0-425e-82ec-fbab68fbe5a5","Type":"ContainerStarted","Data":"f14da10a47ca4c19f73a05f04b76a4e4b4516e5e47d2550183c3feb2e2c1e068"} Oct 29 22:22:28.648401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:28.648002 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:28.648401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:28.648023 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" event={"ID":"a9ca6692-75b0-425e-82ec-fbab68fbe5a5","Type":"ContainerStarted","Data":"7cd247de83012d5806799bc7148f30cb1d73065d960cf6731c6c3645db0019fa"} Oct 29 22:22:28.720544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:28.720491 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" podStartSLOduration=1.720475926 podStartE2EDuration="1.720475926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:28.714650973 +0000 UTC m=+2410.456581647" watchObservedRunningTime="2026-10-29 22:22:28.720475926 +0000 UTC m=+2410.462406599" Oct 29 22:22:38.711243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.711187 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:22:38.793100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.792752 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:22:38.793100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.792894 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:38.831597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.831560 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"e2e-cmp-legacy-53f2ed-provider-cred\"" Oct 29 22:22:38.849778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.849749 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"openshift-service-ca.crt\"" Oct 29 22:22:38.850104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.850063 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"payload-processing-external-model-e2e--e95c4f04-dockercfg-dngqh\"" Oct 29 22:22:38.850344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.850328 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"routing-overlay\"" Oct 29 22:22:38.964153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.911254 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b7c2d5\"" Oct 29 22:22:38.964153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.911366 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"kube-root-ca.crt\"" Oct 29 22:22:38.997099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.984536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:38.997099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.984608 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhpt\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:38.997099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.984676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:38.997099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:38.984837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.098187 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.098253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhpt\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.098314 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.098450 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.102424 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.109323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.103153 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.117522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.117450 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.126767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.126737 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhpt\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.417857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.417773 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:39.661062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.661034 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:22:39.730091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.730042 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:22:39.804930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:39.804897 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" event={"ID":"3c1542ff-88dd-4b00-9d69-1c63451ae575","Type":"ContainerStarted","Data":"966a371e0f7ed01b6e1d738f58d4f2e3e07a98b2f9702451c53d93670368a055"} Oct 29 22:22:40.817190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:40.817140 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" event={"ID":"3c1542ff-88dd-4b00-9d69-1c63451ae575","Type":"ContainerStarted","Data":"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41"} Oct 29 22:22:40.817723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:40.817231 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:40.866191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:40.866136 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" podStartSLOduration=2.8661203889999998 podStartE2EDuration="2.866120389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:40.861604335 +0000 UTC m=+2422.603535009" watchObservedRunningTime="2026-10-29 22:22:40.866120389 +0000 UTC m=+2422.608051063" Oct 29 22:22:44.542510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.542479 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:22:44.548183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.548161 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.566325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.566301 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:22:44.571542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.571507 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-adfb9d\"" Oct 29 22:22:44.571542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.571518 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-adfb9d-dockercfg-xdh6r\"" Oct 29 22:22:44.737679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.737626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.738345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.738226 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5s9d\" (UniqueName: \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.847307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.847215 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5s9d\" (UniqueName: \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.886067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.885820 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.888562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.888532 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:44.902244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:44.902217 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5s9d\" (UniqueName: \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\") pod \"payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:45.165257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.165189 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:45.483778 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:22:45.483746 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeb36ce_ec09_40a5_a268_9d258a946bd1.slice/crio-4a307610a07d5531b31225bdfed7d5d14520f94b854f2c83b5bc2cb8333bda14 WatchSource:0}: Error finding container 4a307610a07d5531b31225bdfed7d5d14520f94b854f2c83b5bc2cb8333bda14: Status 404 returned error can't find the container with id 4a307610a07d5531b31225bdfed7d5d14520f94b854f2c83b5bc2cb8333bda14 Oct 29 22:22:45.484104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.484053 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:22:45.885212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.885120 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" event={"ID":"faeb36ce-ec09-40a5-a268-9d258a946bd1","Type":"ContainerStarted","Data":"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61"} Oct 29 22:22:45.885212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.885173 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" event={"ID":"faeb36ce-ec09-40a5-a268-9d258a946bd1","Type":"ContainerStarted","Data":"4a307610a07d5531b31225bdfed7d5d14520f94b854f2c83b5bc2cb8333bda14"} Oct 29 22:22:45.885745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.885264 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:45.926855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:45.926811 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" podStartSLOduration=1.926798154 podStartE2EDuration="1.926798154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:45.925304963 +0000 UTC m=+2427.667235637" watchObservedRunningTime="2026-10-29 22:22:45.926798154 +0000 UTC m=+2427.668728828" Oct 29 22:22:51.837945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:51.837913 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:22:55.738458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:55.738423 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:22:56.897251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:22:56.897223 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:22:57.975056 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:22:57.975012 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:23:03.581794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.581757 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:23:03.587367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.587349 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.606464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.606439 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:23:03.610932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.610906 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-529e60\"" Oct 29 22:23:03.611379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.611360 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-529e60-dockercfg-5m42k\"" Oct 29 22:23:03.641373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.641350 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjk2\" (UniqueName: \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.641520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.641412 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.812118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.805736 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjk2\" (UniqueName: \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.824436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.824401 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.826194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.825753 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.875773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.875450 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjk2\" (UniqueName: \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\") pod \"payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:03.907291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:03.907210 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:04.255068 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:23:04.255037 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7138bb49_f6a3_46d2_9e84_788a4def2506.slice/crio-743c7a9c64b18948d1e4a0c4d28cac37ec631af06608ead68faa1648cc67a58e WatchSource:0}: Error finding container 743c7a9c64b18948d1e4a0c4d28cac37ec631af06608ead68faa1648cc67a58e: Status 404 returned error can't find the container with id 743c7a9c64b18948d1e4a0c4d28cac37ec631af06608ead68faa1648cc67a58e Oct 29 22:23:04.255587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:04.255563 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:23:05.184888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.184848 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" event={"ID":"7138bb49-f6a3-46d2-9e84-788a4def2506","Type":"ContainerStarted","Data":"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f"} Oct 29 22:23:05.185345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.184911 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" event={"ID":"7138bb49-f6a3-46d2-9e84-788a4def2506","Type":"ContainerStarted","Data":"743c7a9c64b18948d1e4a0c4d28cac37ec631af06608ead68faa1648cc67a58e"} Oct 29 22:23:05.185345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.184943 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:05.226353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.226298 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" podStartSLOduration=2.226282353 podStartE2EDuration="2.226282353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:05.225342789 +0000 UTC m=+2446.967273462" watchObservedRunningTime="2026-10-29 22:23:05.226282353 +0000 UTC m=+2446.968213204" Oct 29 22:23:05.955938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.955909 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr"] Oct 29 22:23:05.961115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.961096 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:05.982479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.982452 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr"] Oct 29 22:23:05.986233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.986193 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:23:05.986377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.986240 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:23:05.988797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.988775 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:23:05.988934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:05.988921 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:23:06.062322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.062289 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8q9v\" (UniqueName: \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.062593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.062573 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.062704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.062687 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.164676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.164642 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.164853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.164744 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.164942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.164883 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8q9v\" (UniqueName: \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.165446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.165419 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.167305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.167284 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.189620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.189591 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8q9v\" (UniqueName: \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\") pod \"maas-api-revoke-sub-d677cb43-gbxlr\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.278143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.278117 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:06.601841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:06.601807 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr"] Oct 29 22:23:06.603174 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:23:06.603143 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3856621b_4b8a_4e64_928a_3394b6a2c54c.slice/crio-9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd WatchSource:0}: Error finding container 9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd: Status 404 returned error can't find the container with id 9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd Oct 29 22:23:07.224100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:07.223716 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" event={"ID":"3856621b-4b8a-4e64-928a-3394b6a2c54c","Type":"ContainerStarted","Data":"60eea0253970e722721b6c2225dfda1d0c0b4190c91a4918a632bb1bb3c6f33c"} Oct 29 22:23:07.224100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:07.223783 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" event={"ID":"3856621b-4b8a-4e64-928a-3394b6a2c54c","Type":"ContainerStarted","Data":"9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd"} Oct 29 22:23:07.305350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:07.296908 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" podStartSLOduration=2.296886669 podStartE2EDuration="2.296886669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:07.278528339 +0000 UTC m=+2449.020459012" watchObservedRunningTime="2026-10-29 22:23:07.296886669 +0000 UTC m=+2449.038817344" Oct 29 22:23:08.236278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:08.236245 4543 generic.go:356] "Generic (PLEG): container finished" podID="3856621b-4b8a-4e64-928a-3394b6a2c54c" containerID="60eea0253970e722721b6c2225dfda1d0c0b4190c91a4918a632bb1bb3c6f33c" exitCode=0 Oct 29 22:23:08.236681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:08.236328 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" event={"ID":"3856621b-4b8a-4e64-928a-3394b6a2c54c","Type":"ContainerDied","Data":"60eea0253970e722721b6c2225dfda1d0c0b4190c91a4918a632bb1bb3c6f33c"} Oct 29 22:23:09.423735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.423713 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:09.551679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.551594 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\") pod \"3856621b-4b8a-4e64-928a-3394b6a2c54c\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " Oct 29 22:23:09.551679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.551639 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\" (UniqueName: \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\") pod \"3856621b-4b8a-4e64-928a-3394b6a2c54c\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " Oct 29 22:23:09.551911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.551763 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\") pod \"3856621b-4b8a-4e64-928a-3394b6a2c54c\" (UID: \"3856621b-4b8a-4e64-928a-3394b6a2c54c\") " Oct 29 22:23:09.552211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.552183 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca" pod "3856621b-4b8a-4e64-928a-3394b6a2c54c" (UID: "3856621b-4b8a-4e64-928a-3394b6a2c54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:23:09.552496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.552480 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:09.553999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.553975 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca" pod "3856621b-4b8a-4e64-928a-3394b6a2c54c" (UID: "3856621b-4b8a-4e64-928a-3394b6a2c54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:23:09.554117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.554012 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v" pod "3856621b-4b8a-4e64-928a-3394b6a2c54c" (UID: "3856621b-4b8a-4e64-928a-3394b6a2c54c"). InnerVolumeSpecName "kube-api-access-d8q9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:09.653851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.653826 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3856621b-4b8a-4e64-928a-3394b6a2c54c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:09.653851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:09.653851 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8q9v\" (UniqueName: \"kubernetes.io/projected/3856621b-4b8a-4e64-928a-3394b6a2c54c-kube-api-access-d8q9v\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:10.253765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:10.253734 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" Oct 29 22:23:10.253949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:10.253787 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr" event={"ID":"3856621b-4b8a-4e64-928a-3394b6a2c54c","Type":"ContainerDied","Data":"9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd"} Oct 29 22:23:10.253949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:10.253842 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4780a6497acb28be49972707d85d053ba350a491e4e29d34a90f0333d431bd" Oct 29 22:23:10.768490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:10.768447 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" containerID="cri-o://ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c" gracePeriod=45 Oct 29 22:23:11.067441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.067416 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:23:11.173729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.173690 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.173932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.173793 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.173932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.173853 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\" (UniqueName: \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.173932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.173892 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.173932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.173924 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:11.174217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.174289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174211 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:11.174289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174232 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\") pod \"ca1055db-7b3e-475a-8c38-5a024c82a40b\" (UID: \"ca1055db-7b3e-475a-8c38-5a024c82a40b\") " Oct 29 22:23:11.174485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174464 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:11.174961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174936 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.175093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174968 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.175093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.174981 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.176330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.176309 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:11.176806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.176783 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:23:11.176806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.176800 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd" pod "ca1055db-7b3e-475a-8c38-5a024c82a40b" (UID: "ca1055db-7b3e-475a-8c38-5a024c82a40b"). InnerVolumeSpecName "kube-api-access-p96vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:11.264342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.264313 4543 generic.go:356] "Generic (PLEG): container finished" podID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerID="ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c" exitCode=0 Oct 29 22:23:11.264514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.264360 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" Oct 29 22:23:11.264514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.264404 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" event={"ID":"ca1055db-7b3e-475a-8c38-5a024c82a40b","Type":"ContainerDied","Data":"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c"} Oct 29 22:23:11.264514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.264450 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc" event={"ID":"ca1055db-7b3e-475a-8c38-5a024c82a40b","Type":"ContainerDied","Data":"996025980c367e953080695a6633e00e7fe716abaae17b475e5ed9169fa2083a"} Oct 29 22:23:11.264514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.264474 4543 scope.go:122] "RemoveContainer" containerID="ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c" Oct 29 22:23:11.278489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.277379 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1055db-7b3e-475a-8c38-5a024c82a40b-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.278489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.277407 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96vd\" (UniqueName: \"kubernetes.io/projected/ca1055db-7b3e-475a-8c38-5a024c82a40b-kube-api-access-p96vd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.278489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.277421 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1055db-7b3e-475a-8c38-5a024c82a40b-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:11.280150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.280132 4543 scope.go:122] "RemoveContainer" containerID="ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c" Oct 29 22:23:11.280453 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:23:11.280433 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c\": container with ID starting with ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c not found: ID does not exist" containerID="ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c" Oct 29 22:23:11.280538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.280464 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c"} err="failed to get container status \"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c\": rpc error: code = NotFound desc = could not find container \"ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c\": container with ID starting with ecdb5a4e9c52844ff0f8d6f9ac8cae5c98b147ebf72b7b5912bcf2ddb529905c not found: ID does not exist" Oct 29 22:23:11.321045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.321020 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:23:11.327034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:11.327012 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8d569e/auth-test-model-8d569e-kserve-584f965ccc-nk2jc"] Oct 29 22:23:12.389190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:12.389157 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" path="/var/lib/kubelet/pods/ca1055db-7b3e-475a-8c38-5a024c82a40b/volumes" Oct 29 22:23:16.197927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:16.197900 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:23:20.957306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:20.957270 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:23:25.958329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:25.958280 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" podUID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" containerName="payload-processing" containerID="cri-o://5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734" gracePeriod=25 Oct 29 22:23:26.280311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.280284 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:23:26.414221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.414192 4543 generic.go:356] "Generic (PLEG): container finished" podID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" containerID="5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734" exitCode=0 Oct 29 22:23:26.414399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.414231 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" Oct 29 22:23:26.414399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.414277 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" event={"ID":"0c4f73c4-ed60-4c5a-865c-44ca60f54603","Type":"ContainerDied","Data":"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734"} Oct 29 22:23:26.414399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.414328 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r" event={"ID":"0c4f73c4-ed60-4c5a-865c-44ca60f54603","Type":"ContainerDied","Data":"8284639e920c2b26830fced9564f912d8e93532a258852b6c11a948b73db7cd1"} Oct 29 22:23:26.414399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.414357 4543 scope.go:122] "RemoveContainer" containerID="5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734" Oct 29 22:23:26.421996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.421969 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\") pod \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " Oct 29 22:23:26.422131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.422025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\") pod \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " Oct 29 22:23:26.422282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.422264 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\") pod \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " Oct 29 22:23:26.422334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.422320 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume" pod "0c4f73c4-ed60-4c5a-865c-44ca60f54603" (UID: "0c4f73c4-ed60-4c5a-865c-44ca60f54603"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:23:26.422400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.422363 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume" pod "0c4f73c4-ed60-4c5a-865c-44ca60f54603" (UID: "0c4f73c4-ed60-4c5a-865c-44ca60f54603"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:23:26.422485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.422468 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\") pod \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\" (UID: \"0c4f73c4-ed60-4c5a-865c-44ca60f54603\") " Oct 29 22:23:26.423133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.423070 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:26.423133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.423114 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c4f73c4-ed60-4c5a-865c-44ca60f54603-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:26.424738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.424708 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials" pod "0c4f73c4-ed60-4c5a-865c-44ca60f54603" (UID: "0c4f73c4-ed60-4c5a-865c-44ca60f54603"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:26.424849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.424768 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc" pod "0c4f73c4-ed60-4c5a-865c-44ca60f54603" (UID: "0c4f73c4-ed60-4c5a-865c-44ca60f54603"). InnerVolumeSpecName "kube-api-access-kfvlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:26.426014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.425988 4543 scope.go:122] "RemoveContainer" containerID="5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734" Oct 29 22:23:26.426349 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:23:26.426328 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734\": container with ID starting with 5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734 not found: ID does not exist" containerID="5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734" Oct 29 22:23:26.426424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.426355 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734"} err="failed to get container status \"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734\": rpc error: code = NotFound desc = could not find container \"5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734\": container with ID starting with 5fc2b6612a97ef0c437e992a9d228e063cf5b7e013892db428da9401c82ed734 not found: ID does not exist" Oct 29 22:23:26.524623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.524559 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfvlc\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-kube-api-access-kfvlc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:26.524623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.524583 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c4f73c4-ed60-4c5a-865c-44ca60f54603-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:23:26.765400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.765374 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:23:26.770152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:26.770129 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-7d5dlrm8r"] Oct 29 22:23:27.962162 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:23:27.962122 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:23:28.389933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:28.389862 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" path="/var/lib/kubelet/pods/0c4f73c4-ed60-4c5a-865c-44ca60f54603/volumes" Oct 29 22:23:30.495250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.495176 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:23:30.496182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496161 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3856621b-4b8a-4e64-928a-3394b6a2c54c" containerName="revoke-keys" Oct 29 22:23:30.496182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496180 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3856621b-4b8a-4e64-928a-3394b6a2c54c" containerName="revoke-keys" Oct 29 22:23:30.496337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496226 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" containerName="payload-processing" Oct 29 22:23:30.496337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496232 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" containerName="payload-processing" Oct 29 22:23:30.496337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496259 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" Oct 29 22:23:30.496337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496264 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" Oct 29 22:23:30.496470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496388 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1055db-7b3e-475a-8c38-5a024c82a40b" containerName="main" Oct 29 22:23:30.496470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496397 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3856621b-4b8a-4e64-928a-3394b6a2c54c" containerName="revoke-keys" Oct 29 22:23:30.496470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.496411 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4f73c4-ed60-4c5a-865c-44ca60f54603" containerName="payload-processing" Oct 29 22:23:30.504616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.504576 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.525572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.525548 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:23:30.531285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.531263 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-1e74c3-openshift-default-dockercfg-s8qfv\"" Oct 29 22:23:30.691240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691196 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691325 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691409 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxsr\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691572 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691647 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691715 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691765 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.691816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.691793 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.792741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792682 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.792741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792726 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.792946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792761 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.792946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792831 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.792946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.792989 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793046 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxsr\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793131 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793214 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793341 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793498 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793627 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.793871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.793772 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.795466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.795438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.795960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.795941 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.820409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.820389 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.820520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.820459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxsr\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\") pod \"e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:30.822926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:30.822909 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:31.155781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.155755 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:23:31.156811 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:23:31.156786 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c04983_e642_4e28_b94c_20474c7ae6bf.slice/crio-90fcdfba149bb0bdbc1266cee0aed34ce00026d1e20777a04b87fa9aac8a4ad6 WatchSource:0}: Error finding container 90fcdfba149bb0bdbc1266cee0aed34ce00026d1e20777a04b87fa9aac8a4ad6: Status 404 returned error can't find the container with id 90fcdfba149bb0bdbc1266cee0aed34ce00026d1e20777a04b87fa9aac8a4ad6 Oct 29 22:23:31.159759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.159726 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:31.159842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.159795 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:31.159842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.159824 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:23:31.461441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.461408 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" event={"ID":"11c04983-e642-4e28-b94c-20474c7ae6bf","Type":"ContainerStarted","Data":"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d"} Oct 29 22:23:31.461590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.461470 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" event={"ID":"11c04983-e642-4e28-b94c-20474c7ae6bf","Type":"ContainerStarted","Data":"90fcdfba149bb0bdbc1266cee0aed34ce00026d1e20777a04b87fa9aac8a4ad6"} Oct 29 22:23:31.509461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.509404 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podStartSLOduration=1.509390113 podStartE2EDuration="1.509390113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:31.507556622 +0000 UTC m=+2473.249487297" watchObservedRunningTime="2026-10-29 22:23:31.509390113 +0000 UTC m=+2473.251320785" Oct 29 22:23:31.823715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:31.823637 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:32.825025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:32.824995 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 29 22:23:32.825429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:32.825048 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 29 22:23:33.823921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:33.823891 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 29 22:23:33.824143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:33.823947 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 29 22:23:34.823705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:34.823671 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 29 22:23:34.824223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:34.823729 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 29 22:23:35.835102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:35.830461 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 29 22:23:35.835102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:35.830524 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 29 22:23:36.823665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:36.823631 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 29 22:23:36.823842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:36.823689 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 29 22:23:37.956063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:37.956028 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:37.956523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:37.956503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:37.957252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:37.957229 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:23:58.045415 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:23:58.043267 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:23:59.123288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.123255 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:23:59.128504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.128484 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.150834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.150810 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:23:59.155869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.155845 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"routing-overlay\"" Oct 29 22:23:59.156018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.155929 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"agc-cred-5664d89903fd\"" Oct 29 22:23:59.156018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.155972 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-e2e--95fdccdd-dockercfg-scf69\"" Oct 29 22:23:59.156018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.155986 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"openshift-service-ca.crt\"" Oct 29 22:23:59.156246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.156046 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a\"" Oct 29 22:23:59.156246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.156050 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"kube-root-ca.crt\"" Oct 29 22:23:59.196528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.196504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjl2r\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.196689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.196671 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.197108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.197030 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.197254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.197235 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.298220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.298195 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.298356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.298241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.298356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.298333 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjl2r\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.298485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.298460 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.299098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.299060 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.299388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.299368 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.301263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.301240 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.323101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.323066 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjl2r\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:23:59.452520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:23:59.452425 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:24:00.043512 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:00.043484 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ca282b_a117_434c_bc71_a72d1fdeb54a.slice/crio-b7b81ef6fa5643603405e02cba4d6fca9bd49874ce9b38bea4d05001d8d91c32 WatchSource:0}: Error finding container b7b81ef6fa5643603405e02cba4d6fca9bd49874ce9b38bea4d05001d8d91c32: Status 404 returned error can't find the container with id b7b81ef6fa5643603405e02cba4d6fca9bd49874ce9b38bea4d05001d8d91c32 Oct 29 22:24:00.044983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.044960 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:24:00.361351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.361261 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:24:00.366460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.366440 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.384470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.384449 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"routing-overlay\"" Oct 29 22:24:00.384635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.384449 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-b9b88c959337\"" Oct 29 22:24:00.384698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.384453 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-695c179c0f51\"" Oct 29 22:24:00.384698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.384465 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"agc-cred-58ea3cdf2b14\"" Oct 29 22:24:00.391818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.391795 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"openshift-service-ca.crt\"" Oct 29 22:24:00.391975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.391829 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"kube-root-ca.crt\"" Oct 29 22:24:00.391975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.391963 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-e2e--d445abcd-dockercfg-f5q7z\"" Oct 29 22:24:00.392202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.392000 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-07e166\"/\"payload-processing-external-model-plugins-e2e-extmod-b-07e166\"" Oct 29 22:24:00.392265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.392199 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:24:00.463378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.463343 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.463517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.463400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.463563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.463524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwdr\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.463696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.463676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.564686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.564659 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.564803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.564695 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.564803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.564751 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwdr\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.564930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.564882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.565590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.565565 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.565725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.565617 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.567746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.567724 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.589401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.589375 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwdr\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\") pod \"payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.683132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.683061 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:00.968746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.968712 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" event={"ID":"00ca282b-a117-434c-bc71-a72d1fdeb54a","Type":"ContainerStarted","Data":"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206"} Oct 29 22:24:00.968925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.968762 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" event={"ID":"00ca282b-a117-434c-bc71-a72d1fdeb54a","Type":"ContainerStarted","Data":"b7b81ef6fa5643603405e02cba4d6fca9bd49874ce9b38bea4d05001d8d91c32"} Oct 29 22:24:00.968925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.968805 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:24:00.999515 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:00.999492 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bbbd28_50cd_4598_bc36_47956400ec28.slice/crio-209f166c97aa8404085b7ed8d84ea9cae992c05e1d0a0bd781577bc130478807 WatchSource:0}: Error finding container 209f166c97aa8404085b7ed8d84ea9cae992c05e1d0a0bd781577bc130478807: Status 404 returned error can't find the container with id 209f166c97aa8404085b7ed8d84ea9cae992c05e1d0a0bd781577bc130478807 Oct 29 22:24:00.999654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:00.999520 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:24:01.012412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:01.012370 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" podStartSLOduration=2.012356326 podStartE2EDuration="2.012356326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:01.010989837 +0000 UTC m=+2502.752920511" watchObservedRunningTime="2026-10-29 22:24:01.012356326 +0000 UTC m=+2502.754287041" Oct 29 22:24:01.985964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:01.985908 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" event={"ID":"62bbbd28-50cd-4598-bc36-47956400ec28","Type":"ContainerStarted","Data":"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1"} Oct 29 22:24:01.986452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:01.986004 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" event={"ID":"62bbbd28-50cd-4598-bc36-47956400ec28","Type":"ContainerStarted","Data":"209f166c97aa8404085b7ed8d84ea9cae992c05e1d0a0bd781577bc130478807"} Oct 29 22:24:01.986452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:01.986049 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:02.028515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:02.028457 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" podStartSLOduration=2.028438773 podStartE2EDuration="2.028438773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:02.027532196 +0000 UTC m=+2503.769462869" watchObservedRunningTime="2026-10-29 22:24:02.028438773 +0000 UTC m=+2503.770369454" Oct 29 22:24:05.600450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:05.600258 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:24:05.619099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:05.618539 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:24:10.601323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.601278 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" podUID="00ca282b-a117-434c-bc71-a72d1fdeb54a" containerName="payload-processing" containerID="cri-o://f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206" gracePeriod=25 Oct 29 22:24:10.891402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.891375 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:24:10.956916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.956881 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\") pod \"00ca282b-a117-434c-bc71-a72d1fdeb54a\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " Oct 29 22:24:10.957256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.957234 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\") pod \"00ca282b-a117-434c-bc71-a72d1fdeb54a\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " Oct 29 22:24:10.957415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.957399 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\") pod \"00ca282b-a117-434c-bc71-a72d1fdeb54a\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " Oct 29 22:24:10.957697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.957678 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\") pod \"00ca282b-a117-434c-bc71-a72d1fdeb54a\" (UID: \"00ca282b-a117-434c-bc71-a72d1fdeb54a\") " Oct 29 22:24:10.960833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.960796 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume" pod "00ca282b-a117-434c-bc71-a72d1fdeb54a" (UID: "00ca282b-a117-434c-bc71-a72d1fdeb54a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:10.960988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.960848 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume" pod "00ca282b-a117-434c-bc71-a72d1fdeb54a" (UID: "00ca282b-a117-434c-bc71-a72d1fdeb54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:10.962997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.962967 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r" pod "00ca282b-a117-434c-bc71-a72d1fdeb54a" (UID: "00ca282b-a117-434c-bc71-a72d1fdeb54a"). InnerVolumeSpecName "kube-api-access-bjl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:10.963425 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:10.963404 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials" pod "00ca282b-a117-434c-bc71-a72d1fdeb54a" (UID: "00ca282b-a117-434c-bc71-a72d1fdeb54a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:11.060227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.060202 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjl2r\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-kube-api-access-bjl2r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:11.060227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.060228 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:11.060385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.060238 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00ca282b-a117-434c-bc71-a72d1fdeb54a-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:11.060385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.060247 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ca282b-a117-434c-bc71-a72d1fdeb54a-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:11.116104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.116028 4543 generic.go:356] "Generic (PLEG): container finished" podID="00ca282b-a117-434c-bc71-a72d1fdeb54a" containerID="f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206" exitCode=0 Oct 29 22:24:11.116104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.116097 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" Oct 29 22:24:11.116248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.116120 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" event={"ID":"00ca282b-a117-434c-bc71-a72d1fdeb54a","Type":"ContainerDied","Data":"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206"} Oct 29 22:24:11.116248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.116201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb" event={"ID":"00ca282b-a117-434c-bc71-a72d1fdeb54a","Type":"ContainerDied","Data":"b7b81ef6fa5643603405e02cba4d6fca9bd49874ce9b38bea4d05001d8d91c32"} Oct 29 22:24:11.116248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.116227 4543 scope.go:122] "RemoveContainer" containerID="f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206" Oct 29 22:24:11.128906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.128890 4543 scope.go:122] "RemoveContainer" containerID="f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206" Oct 29 22:24:11.132922 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:24:11.132894 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206\": container with ID starting with f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206 not found: ID does not exist" containerID="f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206" Oct 29 22:24:11.133039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.132931 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206"} err="failed to get container status \"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206\": rpc error: code = NotFound desc = could not find container \"f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206\": container with ID starting with f81eafef11b2a0f7a8b87bdda3c5528a9de1a674c028e23b232bd96c2b34c206 not found: ID does not exist" Oct 29 22:24:11.194956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.194935 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:24:11.202634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:11.202616 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-74cf5r5bb"] Oct 29 22:24:12.388395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:12.388363 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ca282b-a117-434c-bc71-a72d1fdeb54a" path="/var/lib/kubelet/pods/00ca282b-a117-434c-bc71-a72d1fdeb54a/volumes" Oct 29 22:24:12.997678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:12.997650 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:24:24.269580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.269544 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:24:24.271732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.271702 4543 cpu_manager.go:414] "containerMap: removing container" podUID="00ca282b-a117-434c-bc71-a72d1fdeb54a" containerName="payload-processing" Oct 29 22:24:24.271902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.271885 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ca282b-a117-434c-bc71-a72d1fdeb54a" containerName="payload-processing" Oct 29 22:24:24.272280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.272263 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ca282b-a117-434c-bc71-a72d1fdeb54a" containerName="payload-processing" Oct 29 22:24:24.283152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.283133 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.306209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.304393 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-797e0c-openshift-default-dockercfg-5x88n\"" Oct 29 22:24:24.325139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.321992 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.404691 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtrr\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.404789 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.404883 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.404918 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.405058 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.405204 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.405234 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.405289 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.405578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.405340 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507228 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtrr\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507309 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507395 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507438 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507602 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507732 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507772 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507781 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.507924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.507843 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.508117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.508056 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.508117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.508056 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.508453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.508214 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.508453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.508438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.520363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.520133 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.520589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.520561 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.544604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.544563 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtrr\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.546984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.546926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\") pod \"e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.619302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.619277 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:24.963582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.963554 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:24:24.967243 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:24.967210 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a89316_4d07_4f5a_a551_0c78ede8199d.slice/crio-71ab9058cf008958100204339dd7da430a8e0875c9a0af297a0312590813ba71 WatchSource:0}: Error finding container 71ab9058cf008958100204339dd7da430a8e0875c9a0af297a0312590813ba71: Status 404 returned error can't find the container with id 71ab9058cf008958100204339dd7da430a8e0875c9a0af297a0312590813ba71 Oct 29 22:24:24.970323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.970283 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:24.970409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.970348 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:24.970409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:24.970377 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:25.264918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.264834 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:24:25.270486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.270465 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.287645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.287617 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c1015\"" Oct 29 22:24:25.287794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.287718 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"routing-overlay\"" Oct 29 22:24:25.287873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.287804 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"agc-cred-e54b3fa7dcae\"" Oct 29 22:24:25.288044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.288012 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"payload-processing-external-model-e2e--5de80640-dockercfg-tmpls\"" Oct 29 22:24:25.288213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.288133 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"openshift-service-ca.crt\"" Oct 29 22:24:25.288213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.288207 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"kube-root-ca.crt\"" Oct 29 22:24:25.291122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.291004 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:24:25.320379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.320343 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.320509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.320430 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkh4p\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.320509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.320493 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.320827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.320797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.335070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.335035 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerStarted","Data":"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d"} Oct 29 22:24:25.335230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.335140 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerStarted","Data":"71ab9058cf008958100204339dd7da430a8e0875c9a0af297a0312590813ba71"} Oct 29 22:24:25.486101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.475171 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podStartSLOduration=1.475145693 podStartE2EDuration="1.475145693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:25.4637094 +0000 UTC m=+2527.205640076" watchObservedRunningTime="2026-10-29 22:24:25.475145693 +0000 UTC m=+2527.217076367" Oct 29 22:24:25.502103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.497453 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.502103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.497711 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.502103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.497767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkh4p\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.502103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.497830 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.502103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.499507 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.520146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.515745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.544097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.537655 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.544097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.542552 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkh4p\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.612115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.602005 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:25.706514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:25.697932 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:26.037242 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:26.037213 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca6b60e_058a_4da4_a363_356faa120905.slice/crio-8531b531fbca79e5b5b9d7c72c3b4dbb5bb6a9779d912539ebe8cf2ba502c953 WatchSource:0}: Error finding container 8531b531fbca79e5b5b9d7c72c3b4dbb5bb6a9779d912539ebe8cf2ba502c953: Status 404 returned error can't find the container with id 8531b531fbca79e5b5b9d7c72c3b4dbb5bb6a9779d912539ebe8cf2ba502c953 Oct 29 22:24:26.037405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.037387 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:24:26.345636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.345534 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" event={"ID":"9ca6b60e-058a-4da4-a363-356faa120905","Type":"ContainerStarted","Data":"d1e42af1e310c213d851d78f1b4ac79156c9625e7af2c3d88047deccb8f95c92"} Oct 29 22:24:26.345636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.345586 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" event={"ID":"9ca6b60e-058a-4da4-a363-356faa120905","Type":"ContainerStarted","Data":"8531b531fbca79e5b5b9d7c72c3b4dbb5bb6a9779d912539ebe8cf2ba502c953"} Oct 29 22:24:26.346215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.345886 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:26.393937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.393882 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" podStartSLOduration=1.393864319 podStartE2EDuration="1.393864319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:26.390374831 +0000 UTC m=+2528.132305504" watchObservedRunningTime="2026-10-29 22:24:26.393864319 +0000 UTC m=+2528.135794992" Oct 29 22:24:26.627475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:26.627398 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:27.357232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:27.357198 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:27.358115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:27.358056 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:24:27.963056 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:24:27.963018 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:24:29.940608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:29.940578 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:24:29.945981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:29.945949 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.017102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.016513 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-797e0c-dockercfg-6fjw2\"" Oct 29 22:24:30.020129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.017430 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-797e0c\"" Oct 29 22:24:30.056306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.056266 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.064638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.064608 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5tn\" (UniqueName: \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.142175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.142140 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:24:30.269147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.229504 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5tn\" (UniqueName: \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.269716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.269669 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.285525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.284674 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.291163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.291137 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5tn\" (UniqueName: \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\") pod \"payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.350705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.350679 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:30.665686 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:30.665655 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8493832_1e6d_44e5_a54f_c95d00e6e4c3.slice/crio-0dd078fcddccfe1a65348301a6793dc9570566020682226919f374567175e392 WatchSource:0}: Error finding container 0dd078fcddccfe1a65348301a6793dc9570566020682226919f374567175e392: Status 404 returned error can't find the container with id 0dd078fcddccfe1a65348301a6793dc9570566020682226919f374567175e392 Oct 29 22:24:30.666143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:30.666122 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:24:31.489853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:31.489809 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" event={"ID":"a8493832-1e6d-44e5-a54f-c95d00e6e4c3","Type":"ContainerStarted","Data":"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000"} Oct 29 22:24:31.490502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:31.490473 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" event={"ID":"a8493832-1e6d-44e5-a54f-c95d00e6e4c3","Type":"ContainerStarted","Data":"0dd078fcddccfe1a65348301a6793dc9570566020682226919f374567175e392"} Oct 29 22:24:31.494699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:31.493119 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:31.595483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:31.595420 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" podStartSLOduration=2.595399286 podStartE2EDuration="2.595399286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:31.551282111 +0000 UTC m=+2533.293212786" watchObservedRunningTime="2026-10-29 22:24:31.595399286 +0000 UTC m=+2533.337329960" Oct 29 22:24:35.196881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.196849 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp"] Oct 29 22:24:35.202566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.202547 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.220636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.220612 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp"] Oct 29 22:24:35.226429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.226408 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:24:35.226554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.226449 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:24:35.226554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.226494 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:24:35.229378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.229358 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:24:35.290819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.290795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqlk\" (UniqueName: \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.290990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.290973 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.291110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.291093 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.392574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.392549 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.392680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.392631 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.392805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.392789 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqlk\" (UniqueName: \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.393307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.393263 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.395035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.395014 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.417656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.417628 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqlk\" (UniqueName: \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\") pod \"maas-api-revoke-sub-4e97b061-654pp\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.518193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.518175 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:35.835251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:35.835216 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp"] Oct 29 22:24:36.536407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:36.536313 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" event={"ID":"66131f37-c57d-438f-bd18-19537b1b41e7","Type":"ContainerStarted","Data":"06e731f8e3664246de6616c5fa187d7cc47cfad53badf7eb630c5c5056de8d11"} Oct 29 22:24:36.536407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:36.536370 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" event={"ID":"66131f37-c57d-438f-bd18-19537b1b41e7","Type":"ContainerStarted","Data":"6f2b9fc34b0262bbb6249c7e01880fd924b2304fb120e1a077874fffe45d2afe"} Oct 29 22:24:36.580231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:36.580185 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" podStartSLOduration=1.580171822 podStartE2EDuration="1.580171822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:36.576478992 +0000 UTC m=+2538.318409669" watchObservedRunningTime="2026-10-29 22:24:36.580171822 +0000 UTC m=+2538.322102494" Oct 29 22:24:37.359870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:37.359843 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:24:37.547174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:37.547132 4543 generic.go:356] "Generic (PLEG): container finished" podID="66131f37-c57d-438f-bd18-19537b1b41e7" containerID="06e731f8e3664246de6616c5fa187d7cc47cfad53badf7eb630c5c5056de8d11" exitCode=0 Oct 29 22:24:37.547633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:37.547223 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" event={"ID":"66131f37-c57d-438f-bd18-19537b1b41e7","Type":"ContainerDied","Data":"06e731f8e3664246de6616c5fa187d7cc47cfad53badf7eb630c5c5056de8d11"} Oct 29 22:24:38.725801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.725776 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:38.846573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.846492 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\") pod \"66131f37-c57d-438f-bd18-19537b1b41e7\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " Oct 29 22:24:38.846755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.846602 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\" (UniqueName: \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\") pod \"66131f37-c57d-438f-bd18-19537b1b41e7\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " Oct 29 22:24:38.846755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.846647 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\") pod \"66131f37-c57d-438f-bd18-19537b1b41e7\" (UID: \"66131f37-c57d-438f-bd18-19537b1b41e7\") " Oct 29 22:24:38.847031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.847001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca" pod "66131f37-c57d-438f-bd18-19537b1b41e7" (UID: "66131f37-c57d-438f-bd18-19537b1b41e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:38.847489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.847464 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:38.848952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.848923 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca" pod "66131f37-c57d-438f-bd18-19537b1b41e7" (UID: "66131f37-c57d-438f-bd18-19537b1b41e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:24:38.849041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.848981 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk" pod "66131f37-c57d-438f-bd18-19537b1b41e7" (UID: "66131f37-c57d-438f-bd18-19537b1b41e7"). InnerVolumeSpecName "kube-api-access-spqlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:38.949181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.949151 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqlk\" (UniqueName: \"kubernetes.io/projected/66131f37-c57d-438f-bd18-19537b1b41e7-kube-api-access-spqlk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:38.949181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:38.949181 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66131f37-c57d-438f-bd18-19537b1b41e7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:39.576494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:39.576466 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" Oct 29 22:24:39.576668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:39.576502 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp" event={"ID":"66131f37-c57d-438f-bd18-19537b1b41e7","Type":"ContainerDied","Data":"6f2b9fc34b0262bbb6249c7e01880fd924b2304fb120e1a077874fffe45d2afe"} Oct 29 22:24:39.576668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:39.576566 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f2b9fc34b0262bbb6249c7e01880fd924b2304fb120e1a077874fffe45d2afe" Oct 29 22:24:43.511595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:43.511567 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:24:47.650479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.650448 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr"] Oct 29 22:24:47.651641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.651617 4543 cpu_manager.go:414] "containerMap: removing container" podUID="66131f37-c57d-438f-bd18-19537b1b41e7" containerName="revoke-keys" Oct 29 22:24:47.651641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.651640 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66131f37-c57d-438f-bd18-19537b1b41e7" containerName="revoke-keys" Oct 29 22:24:47.651936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.651920 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="66131f37-c57d-438f-bd18-19537b1b41e7" containerName="revoke-keys" Oct 29 22:24:47.656053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.656032 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.674376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.674354 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:24:47.675331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.675308 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr"] Oct 29 22:24:47.676125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.676101 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqnz\" (UniqueName: \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.676187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.676174 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.676411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.676392 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.680278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.680256 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:24:47.680278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.680273 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:24:47.680460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.680301 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:24:47.777991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.777955 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.778178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.778147 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqnz\" (UniqueName: \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.778234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.778206 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.778868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.778839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.780595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.780573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.803144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.803118 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqnz\" (UniqueName: \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\") pod \"maas-api-revoke-sub-6b974f38-f27dr\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:47.973518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:47.973489 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:48.043170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.043135 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k"] Oct 29 22:24:48.049976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.049947 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl"] Oct 29 22:24:48.056647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.056623 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516407b8-z4c7k"] Oct 29 22:24:48.063724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.063697 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6002b493-9bksl"] Oct 29 22:24:48.298622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.298593 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr"] Oct 29 22:24:48.298956 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:48.298928 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e77e9dc_88d1_474c_840f_e278269f1099.slice/crio-5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac WatchSource:0}: Error finding container 5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac: Status 404 returned error can't find the container with id 5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac Oct 29 22:24:48.389465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.389441 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc79acd-2e04-4adf-a239-4118d09d5c59" path="/var/lib/kubelet/pods/2dc79acd-2e04-4adf-a239-4118d09d5c59/volumes" Oct 29 22:24:48.389893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.389877 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a612e65-485f-41a4-a367-003f3e594478" path="/var/lib/kubelet/pods/8a612e65-485f-41a4-a367-003f3e594478/volumes" Oct 29 22:24:48.662640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:48.662608 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" event={"ID":"2e77e9dc-88d1-474c-840f-e278269f1099","Type":"ContainerStarted","Data":"5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac"} Oct 29 22:24:49.677139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:49.677108 4543 generic.go:356] "Generic (PLEG): container finished" podID="2e77e9dc-88d1-474c-840f-e278269f1099" containerID="fb7e9dab1c7eec8ecc92b7bb225f82dc89496f9d8ca5a782a1a735019adb819e" exitCode=0 Oct 29 22:24:49.677513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:49.677191 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" event={"ID":"2e77e9dc-88d1-474c-840f-e278269f1099","Type":"ContainerDied","Data":"fb7e9dab1c7eec8ecc92b7bb225f82dc89496f9d8ca5a782a1a735019adb819e"} Oct 29 22:24:50.660710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.660679 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:24:50.666163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.666142 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.689633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.689600 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:24:50.692650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.692607 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ff15ac93a07f\"" Oct 29 22:24:50.693520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.693486 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l8qmt\"" Oct 29 22:24:50.694010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.693985 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 22:24:50.694133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.693989 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 22:24:50.694342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.694322 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 22:24:50.696374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.696332 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 22:24:50.756274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.756243 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.756619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.756594 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.756802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.756783 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.756884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.756839 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxln\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.891221 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.891417 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.891458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxln\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.891524 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.892721 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.909453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.893303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.963147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.955650 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:50.963147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:50.956330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxln\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\") pod \"payload-processing-external-model-695ccd988f-9bmkp\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:51.020137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.014002 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:51.149878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.149843 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:51.312215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.312150 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\" (UniqueName: \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\") pod \"2e77e9dc-88d1-474c-840f-e278269f1099\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " Oct 29 22:24:51.312215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.312190 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\") pod \"2e77e9dc-88d1-474c-840f-e278269f1099\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " Oct 29 22:24:51.312535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.312447 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\") pod \"2e77e9dc-88d1-474c-840f-e278269f1099\" (UID: \"2e77e9dc-88d1-474c-840f-e278269f1099\") " Oct 29 22:24:51.312813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.312791 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca" pod "2e77e9dc-88d1-474c-840f-e278269f1099" (UID: "2e77e9dc-88d1-474c-840f-e278269f1099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:51.313170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.313150 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.314502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.314481 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca" pod "2e77e9dc-88d1-474c-840f-e278269f1099" (UID: "2e77e9dc-88d1-474c-840f-e278269f1099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:24:51.314573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.314544 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz" pod "2e77e9dc-88d1-474c-840f-e278269f1099" (UID: "2e77e9dc-88d1-474c-840f-e278269f1099"). InnerVolumeSpecName "kube-api-access-7bqnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:51.414347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.414311 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqnz\" (UniqueName: \"kubernetes.io/projected/2e77e9dc-88d1-474c-840f-e278269f1099-kube-api-access-7bqnz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.414347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.414336 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e77e9dc-88d1-474c-840f-e278269f1099-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:24:51.442177 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:51.442151 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0259ea5_a981_499d_985c_00c071d8f23a.slice/crio-6a6e86aaf531073389f13f7b4d99b09b4966af59f8632e73feb06212fdf96652 WatchSource:0}: Error finding container 6a6e86aaf531073389f13f7b4d99b09b4966af59f8632e73feb06212fdf96652: Status 404 returned error can't find the container with id 6a6e86aaf531073389f13f7b4d99b09b4966af59f8632e73feb06212fdf96652 Oct 29 22:24:51.442714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.442639 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:24:51.700929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.700853 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" event={"ID":"f0259ea5-a981-499d-985c-00c071d8f23a","Type":"ContainerStarted","Data":"91f7470d42c7539e57b11fe486d46c8fd8cfd6ecdfcc8a802168135f37d72f4b"} Oct 29 22:24:51.701369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.701282 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" event={"ID":"f0259ea5-a981-499d-985c-00c071d8f23a","Type":"ContainerStarted","Data":"6a6e86aaf531073389f13f7b4d99b09b4966af59f8632e73feb06212fdf96652"} Oct 29 22:24:51.701369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.701325 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:24:51.704250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.704218 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" event={"ID":"2e77e9dc-88d1-474c-840f-e278269f1099","Type":"ContainerDied","Data":"5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac"} Oct 29 22:24:51.704386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.704319 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd246dc348495fe4c66d31915f70d3ae43e38b18a09afc7a0e562b26969b2ac" Oct 29 22:24:51.704451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.704404 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr" Oct 29 22:24:51.748634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:51.748588 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" podStartSLOduration=1.748573521 podStartE2EDuration="1.748573521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:51.743155254 +0000 UTC m=+2553.485085988" watchObservedRunningTime="2026-10-29 22:24:51.748573521 +0000 UTC m=+2553.490504193" Oct 29 22:24:55.636122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:55.636092 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:24:57.840441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.840405 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:24:57.841437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.841410 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2e77e9dc-88d1-474c-840f-e278269f1099" containerName="revoke-keys" Oct 29 22:24:57.841437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.841431 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e77e9dc-88d1-474c-840f-e278269f1099" containerName="revoke-keys" Oct 29 22:24:57.841629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.841616 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e77e9dc-88d1-474c-840f-e278269f1099" containerName="revoke-keys" Oct 29 22:24:57.851428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.851402 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.879044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.879015 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:24:57.944729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.944689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.944909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.944890 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945000 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945273 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945381 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.945412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:57.945403 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:57.962714 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:24:57.962687 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:24:58.036929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.036904 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng"] Oct 29 22:24:58.042163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.042138 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8"] Oct 29 22:24:58.047338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047309 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047342 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047398 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047407 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21754e44-jnpng"] Oct 29 22:24:58.047628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047610 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047705 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047736 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047815 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047836 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.047963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047932 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.048050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047968 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.048050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.047997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.048200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.048054 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.048654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.048629 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.048731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.048689 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.050584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.050552 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.050789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.050770 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.052215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.052196 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3336a63-n7cg8"] Oct 29 22:24:58.074580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.074560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.074780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.074759 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\") pod \"maas-default-gateway-openshift-default-6f54779779-tv5rf\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.179684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.179615 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:58.388593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.388560 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527dbae9-e173-42df-8275-269f2e8b3f7f" path="/var/lib/kubelet/pods/527dbae9-e173-42df-8275-269f2e8b3f7f/volumes" Oct 29 22:24:58.389052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.389036 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79dd9f9-69fa-434d-9918-39f6b2bec85c" path="/var/lib/kubelet/pods/a79dd9f9-69fa-434d-9918-39f6b2bec85c/volumes" Oct 29 22:24:58.514453 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:24:58.514431 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89a9a05_ea8c_432f_90f9_df3f6bf20645.slice/crio-e77d8a85ce48188f77d513181e1299fc1e8b3c5ba29c4c2751f93b71424deafe WatchSource:0}: Error finding container e77d8a85ce48188f77d513181e1299fc1e8b3c5ba29c4c2751f93b71424deafe: Status 404 returned error can't find the container with id e77d8a85ce48188f77d513181e1299fc1e8b3c5ba29c4c2751f93b71424deafe Oct 29 22:24:58.515208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.515187 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:24:58.517631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.517595 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:58.517752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.517673 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:58.517752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.517717 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:24:58.789566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.789469 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" event={"ID":"a89a9a05-ea8c-432f-90f9-df3f6bf20645","Type":"ContainerStarted","Data":"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b"} Oct 29 22:24:58.789566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.789529 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" event={"ID":"a89a9a05-ea8c-432f-90f9-df3f6bf20645","Type":"ContainerStarted","Data":"e77d8a85ce48188f77d513181e1299fc1e8b3c5ba29c4c2751f93b71424deafe"} Oct 29 22:24:58.836951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.836894 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podStartSLOduration=1.8368783560000002 podStartE2EDuration="1.836878356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:58.834619428 +0000 UTC m=+2560.576550100" watchObservedRunningTime="2026-10-29 22:24:58.836878356 +0000 UTC m=+2560.578809030" Oct 29 22:24:58.964548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:58.964511 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:24:59.180713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:59.180602 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:24:59.182992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:59.182958 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:24:59.183185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:24:59.183014 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:00.181127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:00.181090 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:00.181598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:00.181147 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:00.650147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:00.649920 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" podUID="f0259ea5-a981-499d-985c-00c071d8f23a" containerName="payload-processing" containerID="cri-o://91f7470d42c7539e57b11fe486d46c8fd8cfd6ecdfcc8a802168135f37d72f4b" gracePeriod=25 Oct 29 22:25:00.857653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:00.857617 4543 generic.go:356] "Generic (PLEG): container finished" podID="f0259ea5-a981-499d-985c-00c071d8f23a" containerID="91f7470d42c7539e57b11fe486d46c8fd8cfd6ecdfcc8a802168135f37d72f4b" exitCode=0 Oct 29 22:25:00.857947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:00.857917 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" event={"ID":"f0259ea5-a981-499d-985c-00c071d8f23a","Type":"ContainerDied","Data":"91f7470d42c7539e57b11fe486d46c8fd8cfd6ecdfcc8a802168135f37d72f4b"} Oct 29 22:25:01.026593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.026568 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:25:01.122754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.122726 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\") pod \"f0259ea5-a981-499d-985c-00c071d8f23a\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " Oct 29 22:25:01.122754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.122762 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\") pod \"f0259ea5-a981-499d-985c-00c071d8f23a\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " Oct 29 22:25:01.123122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.122858 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\") pod \"f0259ea5-a981-499d-985c-00c071d8f23a\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " Oct 29 22:25:01.123122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.122942 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\") pod \"f0259ea5-a981-499d-985c-00c071d8f23a\" (UID: \"f0259ea5-a981-499d-985c-00c071d8f23a\") " Oct 29 22:25:01.123209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.123175 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume" pod "f0259ea5-a981-499d-985c-00c071d8f23a" (UID: "f0259ea5-a981-499d-985c-00c071d8f23a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:01.123853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.123584 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume" pod "f0259ea5-a981-499d-985c-00c071d8f23a" (UID: "f0259ea5-a981-499d-985c-00c071d8f23a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:01.123853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.123821 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:01.123853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.123836 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0259ea5-a981-499d-985c-00c071d8f23a-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:01.125424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.125398 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials" pod "f0259ea5-a981-499d-985c-00c071d8f23a" (UID: "f0259ea5-a981-499d-985c-00c071d8f23a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:01.159412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.159278 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln" pod "f0259ea5-a981-499d-985c-00c071d8f23a" (UID: "f0259ea5-a981-499d-985c-00c071d8f23a"). InnerVolumeSpecName "kube-api-access-jdxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:01.180019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.179993 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:01.180169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.180041 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:01.225469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.225440 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxln\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-kube-api-access-jdxln\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:01.225800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.225475 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0259ea5-a981-499d-985c-00c071d8f23a-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:01.871354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.871322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" Oct 29 22:25:01.871354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.871338 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp" event={"ID":"f0259ea5-a981-499d-985c-00c071d8f23a","Type":"ContainerDied","Data":"6a6e86aaf531073389f13f7b4d99b09b4966af59f8632e73feb06212fdf96652"} Oct 29 22:25:01.871596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.871400 4543 scope.go:122] "RemoveContainer" containerID="91f7470d42c7539e57b11fe486d46c8fd8cfd6ecdfcc8a802168135f37d72f4b" Oct 29 22:25:01.941124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.937717 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:25:01.960139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:01.958724 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-9bmkp"] Oct 29 22:25:02.180600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:02.180517 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:02.180600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:02.180574 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:02.389344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:02.389311 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0259ea5-a981-499d-985c-00c071d8f23a" path="/var/lib/kubelet/pods/f0259ea5-a981-499d-985c-00c071d8f23a/volumes" Oct 29 22:25:03.180313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:03.180281 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:03.180493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:03.180346 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:04.025142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.015674 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" podUID="3c1542ff-88dd-4b00-9d69-1c63451ae575" containerName="payload-processing" containerID="cri-o://f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41" gracePeriod=25 Oct 29 22:25:04.181264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.181204 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:04.181264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.181254 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:04.580488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.580420 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:25:04.715229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715196 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\") pod \"3c1542ff-88dd-4b00-9d69-1c63451ae575\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " Oct 29 22:25:04.715417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715300 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\") pod \"3c1542ff-88dd-4b00-9d69-1c63451ae575\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " Oct 29 22:25:04.715417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715359 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\") pod \"3c1542ff-88dd-4b00-9d69-1c63451ae575\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " Oct 29 22:25:04.715590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715570 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\") pod \"3c1542ff-88dd-4b00-9d69-1c63451ae575\" (UID: \"3c1542ff-88dd-4b00-9d69-1c63451ae575\") " Oct 29 22:25:04.715726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715698 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume" pod "3c1542ff-88dd-4b00-9d69-1c63451ae575" (UID: "3c1542ff-88dd-4b00-9d69-1c63451ae575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:04.715777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.715734 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume" pod "3c1542ff-88dd-4b00-9d69-1c63451ae575" (UID: "3c1542ff-88dd-4b00-9d69-1c63451ae575"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:04.716309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.716290 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:04.716373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.716311 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c1542ff-88dd-4b00-9d69-1c63451ae575-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:04.717733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.717705 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials" pod "3c1542ff-88dd-4b00-9d69-1c63451ae575" (UID: "3c1542ff-88dd-4b00-9d69-1c63451ae575"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:04.717950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.717928 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt" pod "3c1542ff-88dd-4b00-9d69-1c63451ae575" (UID: "3c1542ff-88dd-4b00-9d69-1c63451ae575"). InnerVolumeSpecName "kube-api-access-rvhpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:04.853169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.840132 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:04.853169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.840192 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvhpt\" (UniqueName: \"kubernetes.io/projected/3c1542ff-88dd-4b00-9d69-1c63451ae575-kube-api-access-rvhpt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:04.865107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.860580 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879542 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f0259ea5-a981-499d-985c-00c071d8f23a" containerName="payload-processing" Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879571 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0259ea5-a981-499d-985c-00c071d8f23a" containerName="payload-processing" Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879628 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3c1542ff-88dd-4b00-9d69-1c63451ae575" containerName="payload-processing" Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879636 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1542ff-88dd-4b00-9d69-1c63451ae575" containerName="payload-processing" Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879860 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1542ff-88dd-4b00-9d69-1c63451ae575" containerName="payload-processing" Oct 29 22:25:04.881133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.879883 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0259ea5-a981-499d-985c-00c071d8f23a" containerName="payload-processing" Oct 29 22:25:04.981642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.978762 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:04.983283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.982901 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:25:04.997464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.995385 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"agc-cred-329069323922\"" Oct 29 22:25:04.997464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:04.995486 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b7c2d5\"/\"payload-processing-external-model-e2e--e95c4f04-dockercfg-4zrsl\"" Oct 29 22:25:05.054383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.054331 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.054859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.054538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.054859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.054650 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.054859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.054732 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6cn\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.157608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.157504 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6cn\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.158057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.158032 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.158403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.158383 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.158614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.158597 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.159963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.159927 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.160265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.160237 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.161095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.161051 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.170704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.170677 4543 generic.go:356] "Generic (PLEG): container finished" podID="3c1542ff-88dd-4b00-9d69-1c63451ae575" containerID="f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41" exitCode=0 Oct 29 22:25:05.170820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.170736 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" Oct 29 22:25:05.170820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.170762 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" event={"ID":"3c1542ff-88dd-4b00-9d69-1c63451ae575","Type":"ContainerDied","Data":"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41"} Oct 29 22:25:05.170946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.170831 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w" event={"ID":"3c1542ff-88dd-4b00-9d69-1c63451ae575","Type":"ContainerDied","Data":"966a371e0f7ed01b6e1d738f58d4f2e3e07a98b2f9702451c53d93670368a055"} Oct 29 22:25:05.170946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.170863 4543 scope.go:122] "RemoveContainer" containerID="f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41" Oct 29 22:25:05.180521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.180499 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:05.180652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.180560 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:05.187365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.187340 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6cn\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\") pod \"payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:05.191206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.191190 4543 scope.go:122] "RemoveContainer" containerID="f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41" Oct 29 22:25:05.191529 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:05.191509 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41\": container with ID starting with f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41 not found: ID does not exist" containerID="f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41" Oct 29 22:25:05.191601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.191539 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41"} err="failed to get container status \"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41\": rpc error: code = NotFound desc = could not find container \"f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41\": container with ID starting with f2f556020d0f09cdc55960b9fd31446ea9384b38044a978052b7560786d58d41 not found: ID does not exist" Oct 29 22:25:05.245220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.245191 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:25:05.254183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.254150 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-6d9bnmf8w"] Oct 29 22:25:05.307030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:05.307002 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:06.076525 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:25:06.076488 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc77a3b0_8321_4632_aa0b_b2d82b66953b.slice/crio-fd0910e0d8e5d1004bcca817a7108127f0a025f64678729cd5314886fb73c2bd WatchSource:0}: Error finding container fd0910e0d8e5d1004bcca817a7108127f0a025f64678729cd5314886fb73c2bd: Status 404 returned error can't find the container with id fd0910e0d8e5d1004bcca817a7108127f0a025f64678729cd5314886fb73c2bd Oct 29 22:25:06.078810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.078782 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:25:06.180966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.180938 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:06.181123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.180985 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:06.186036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.186009 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" event={"ID":"dc77a3b0-8321-4632-aa0b-b2d82b66953b","Type":"ContainerStarted","Data":"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924"} Oct 29 22:25:06.186199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.186066 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" event={"ID":"dc77a3b0-8321-4632-aa0b-b2d82b66953b","Type":"ContainerStarted","Data":"fd0910e0d8e5d1004bcca817a7108127f0a025f64678729cd5314886fb73c2bd"} Oct 29 22:25:06.186199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.186144 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:06.230641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.230597 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" podStartSLOduration=2.230584461 podStartE2EDuration="2.230584461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:06.227642492 +0000 UTC m=+2567.969573166" watchObservedRunningTime="2026-10-29 22:25:06.230584461 +0000 UTC m=+2567.972515136" Oct 29 22:25:06.389859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:06.389782 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1542ff-88dd-4b00-9d69-1c63451ae575" path="/var/lib/kubelet/pods/3c1542ff-88dd-4b00-9d69-1c63451ae575/volumes" Oct 29 22:25:07.180291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:07.180258 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 29 22:25:07.180741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:07.180314 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 29 22:25:08.876159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.876122 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:08.932762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.932736 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:08.933332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.933246 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:08.950107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.949279 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-da41191393ea\"" Oct 29 22:25:08.950107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.950045 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-f7msq\"" Oct 29 22:25:08.950332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.950306 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 22:25:08.950578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.950459 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 22:25:08.960881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.960858 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 22:25:08.961001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.960949 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 22:25:08.975267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.975223 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdwx\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:08.975407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.975292 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:08.975481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.975406 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:08.975730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:08.975699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.077772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.077731 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.077999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.077984 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdwx\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.078050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.078023 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.078144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.078066 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.078576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.078544 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.078738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.078721 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.080815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.080789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.102924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.102886 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdwx\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\") pod \"payload-processing-external-model-589b4c9d64-697cr\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.180929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.180848 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:25:09.181098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.180918 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:09.270414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.270383 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:09.602707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:09.598997 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:09.606491 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:25:09.606444 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5139d749_7a35_4beb_ac83_76efbab179ce.slice/crio-c1c28695fc3308572c43ec0d4daf7aeee8f5a7a8b7c44b4d3e686eb3a937bcf1 WatchSource:0}: Error finding container c1c28695fc3308572c43ec0d4daf7aeee8f5a7a8b7c44b4d3e686eb3a937bcf1: Status 404 returned error can't find the container with id c1c28695fc3308572c43ec0d4daf7aeee8f5a7a8b7c44b4d3e686eb3a937bcf1 Oct 29 22:25:10.222219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.222190 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:25:10.222715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.222252 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:10.231446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.231415 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" event={"ID":"5139d749-7a35-4beb-ac83-76efbab179ce","Type":"ContainerStarted","Data":"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e"} Oct 29 22:25:10.231611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.231467 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" event={"ID":"5139d749-7a35-4beb-ac83-76efbab179ce","Type":"ContainerStarted","Data":"c1c28695fc3308572c43ec0d4daf7aeee8f5a7a8b7c44b4d3e686eb3a937bcf1"} Oct 29 22:25:10.231611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.231519 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:10.281431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:10.281360 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" podStartSLOduration=2.281341438 podStartE2EDuration="2.281341438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:10.272493204 +0000 UTC m=+2572.014423877" watchObservedRunningTime="2026-10-29 22:25:10.281341438 +0000 UTC m=+2572.023272124" Oct 29 22:25:11.290100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:11.275367 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-tv5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:25:11.290100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:11.275427 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:11.511783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:11.511750 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:25:11.512344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:11.512265 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:25:11.514300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:11.514278 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:25:12.892162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.892130 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:25:12.905415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.905384 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.923328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.921553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:25:12.997950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.997920 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwmj\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.997961 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998204 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998296 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998321 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998484 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998668 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:12.998762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:12.998706 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.099842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.099808 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwmj\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.099968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.099855 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.099988 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100062 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100149 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100185 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100317 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100346 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100423 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100610 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100679 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.100776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.100754 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.101377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.101349 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.102575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.102557 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.102715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.102696 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.140804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.124533 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.140804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.125559 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwmj\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\") pod \"maas-default-gateway-openshift-default-6f54779779-dfq6q\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.234788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.234512 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:13.658166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.658051 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:13.763102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.762198 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.766683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.765559 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:13.779998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.779959 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:13.780151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.780035 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:13.780151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.780095 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:25:13.781402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.781381 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-96cc362d023d\"" Oct 29 22:25:13.781634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.781621 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"routing-overlay\"" Oct 29 22:25:13.781818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.781803 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"openshift-service-ca.crt\"" Oct 29 22:25:13.782038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.782020 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e7b56d\"" Oct 29 22:25:13.782273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.782257 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"kube-root-ca.crt\"" Oct 29 22:25:13.795151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.793839 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"payload-processing-external-model-e2e--885c5d1d-dockercfg-58nd4\"" Oct 29 22:25:13.827136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.817130 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:25:13.827136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.826454 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:13.847903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.847872 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.848217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.848192 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9d4\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.848318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.848237 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.848554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.848535 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.869994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.868013 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:25:13.869994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.868262 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" containerID="cri-o://94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024" gracePeriod=30 Oct 29 22:25:13.950746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.950671 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.951152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.950756 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.951152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.950984 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9d4\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.951152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.951013 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.951552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.951522 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.951827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.951803 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.953824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.953790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:13.975947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:13.975926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9d4\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:14.089767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:14.089736 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:14.494100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:14.493775 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" event={"ID":"ff40db28-0d24-4cca-a739-5d0c3448503e","Type":"ContainerStarted","Data":"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4"} Oct 29 22:25:14.494100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:14.493842 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" event={"ID":"ff40db28-0d24-4cca-a739-5d0c3448503e","Type":"ContainerStarted","Data":"dcb62d5d65a9bf8c2ab71aa9c2eef24154b68606872a0db68c5f8dac624ae5c5"} Oct 29 22:25:14.563834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:14.558828 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podStartSLOduration=2.5588072779999997 podStartE2EDuration="2.558807278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:14.548466701 +0000 UTC m=+2576.290397373" watchObservedRunningTime="2026-10-29 22:25:14.558807278 +0000 UTC m=+2576.300737952" Oct 29 22:25:14.645299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:14.639802 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:15.253100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.241465 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:15.260239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.260201 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:15.260448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.260267 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:15.506426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.506339 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" event={"ID":"c715da87-029e-4d30-a48b-383918d23e0d","Type":"ContainerStarted","Data":"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03"} Oct 29 22:25:15.506426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.506391 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" event={"ID":"c715da87-029e-4d30-a48b-383918d23e0d","Type":"ContainerStarted","Data":"2dfdd1db3c4e4c31026b67a489a1d6f25672020a8a9c5a8a48a6f6278b1ac666"} Oct 29 22:25:15.506684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.506609 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:15.560769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:15.560708 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" podStartSLOduration=2.560690763 podStartE2EDuration="2.560690763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:15.555857575 +0000 UTC m=+2577.297788248" watchObservedRunningTime="2026-10-29 22:25:15.560690763 +0000 UTC m=+2577.302621437" Oct 29 22:25:16.253100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:16.251383 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:16.253100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:16.251440 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:17.249101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:17.235514 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:17.249101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:17.235577 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:17.263099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:17.259787 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:25:18.252100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.251487 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:18.252100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.251547 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:18.687064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.686971 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:25:18.695389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.695359 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.699659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.699631 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:25:18.711651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.711631 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"openshift-service-ca.crt\"" Oct 29 22:25:18.722779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.722760 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"agc-cred-d3185baddced\"" Oct 29 22:25:18.722862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.722763 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"kube-root-ca.crt\"" Oct 29 22:25:18.722862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.722820 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"routing-overlay\"" Oct 29 22:25:18.722862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.722821 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-e2e--605ba260-dockercfg-56bp5\"" Oct 29 22:25:18.722970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.722849 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9ed94\"" Oct 29 22:25:18.826098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.820006 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.826098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.821328 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.826098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.821482 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvk85\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.826098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.821528 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.841099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.835123 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" podUID="5139d749-7a35-4beb-ac83-76efbab179ce" containerName="payload-processing" containerID="cri-o://85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e" gracePeriod=25 Oct 29 22:25:18.978149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.928610 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.978149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.928768 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvk85\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.978149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.928817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.978149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.928975 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:18.978149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.957406 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:19.011097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:18.983677 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:19.057846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.057781 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:19.061246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.061064 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvk85\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:19.253153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.239468 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:19.253153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.239528 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:19.306993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.306967 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:25:19.314429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.314403 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:19.365891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.365848 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.365985 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.366145 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.366224 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.366316 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.366374 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.366542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.366454 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.375201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.375152 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.375313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.375243 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\") pod \"bc224878-39b1-4043-9d1c-a2f6b607803a\" (UID: \"bc224878-39b1-4043-9d1c-a2f6b607803a\") " Oct 29 22:25:19.376713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.376655 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:19.377163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.377112 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:19.377550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.377374 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:19.378033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.377797 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:19.378033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.377813 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:19.378282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.378252 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:19.385546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.385522 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.386533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.386507 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:25:19.390512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.390389 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc" pod "bc224878-39b1-4043-9d1c-a2f6b607803a" (UID: "bc224878-39b1-4043-9d1c-a2f6b607803a"). InnerVolumeSpecName "kube-api-access-59qrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.475767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.474215 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483887 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc224878-39b1-4043-9d1c-a2f6b607803a-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483916 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483929 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483942 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59qrc\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-kube-api-access-59qrc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483960 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483971 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483984 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.483995 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.484024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.484014 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc224878-39b1-4043-9d1c-a2f6b607803a-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.584884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.584789 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\") pod \"5139d749-7a35-4beb-ac83-76efbab179ce\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " Oct 29 22:25:19.585231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.585201 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume" pod "5139d749-7a35-4beb-ac83-76efbab179ce" (UID: "5139d749-7a35-4beb-ac83-76efbab179ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:19.585391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.585370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\") pod \"5139d749-7a35-4beb-ac83-76efbab179ce\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " Oct 29 22:25:19.585489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.585475 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\") pod \"5139d749-7a35-4beb-ac83-76efbab179ce\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " Oct 29 22:25:19.585736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.585716 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\") pod \"5139d749-7a35-4beb-ac83-76efbab179ce\" (UID: \"5139d749-7a35-4beb-ac83-76efbab179ce\") " Oct 29 22:25:19.586168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.585884 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume" pod "5139d749-7a35-4beb-ac83-76efbab179ce" (UID: "5139d749-7a35-4beb-ac83-76efbab179ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:19.586708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.586687 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.586804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.586713 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5139d749-7a35-4beb-ac83-76efbab179ce-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.588582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.588553 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx" pod "5139d749-7a35-4beb-ac83-76efbab179ce" (UID: "5139d749-7a35-4beb-ac83-76efbab179ce"). InnerVolumeSpecName "kube-api-access-xjdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.588661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.588606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials" pod "5139d749-7a35-4beb-ac83-76efbab179ce" (UID: "5139d749-7a35-4beb-ac83-76efbab179ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.689270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689237 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdwx\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-kube-api-access-xjdwx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.689270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689268 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5139d749-7a35-4beb-ac83-76efbab179ce-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.690123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689839 4543 generic.go:356] "Generic (PLEG): container finished" podID="5139d749-7a35-4beb-ac83-76efbab179ce" containerID="85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e" exitCode=0 Oct 29 22:25:19.690123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689874 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" event={"ID":"5139d749-7a35-4beb-ac83-76efbab179ce","Type":"ContainerDied","Data":"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e"} Oct 29 22:25:19.690123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689895 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" Oct 29 22:25:19.690123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689940 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-589b4c9d64-697cr" event={"ID":"5139d749-7a35-4beb-ac83-76efbab179ce","Type":"ContainerDied","Data":"c1c28695fc3308572c43ec0d4daf7aeee8f5a7a8b7c44b4d3e686eb3a937bcf1"} Oct 29 22:25:19.690123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.689959 4543 scope.go:122] "RemoveContainer" containerID="85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e" Oct 29 22:25:19.691863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.691838 4543 generic.go:356] "Generic (PLEG): container finished" podID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerID="94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024" exitCode=0 Oct 29 22:25:19.692312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.692285 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" event={"ID":"bc224878-39b1-4043-9d1c-a2f6b607803a","Type":"ContainerDied","Data":"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024"} Oct 29 22:25:19.692423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.692343 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" event={"ID":"bc224878-39b1-4043-9d1c-a2f6b607803a","Type":"ContainerDied","Data":"62f062add873d6d2513aee4fd68b697cd8021c7a41f6b5a3bbdc8a69412d7137"} Oct 29 22:25:19.692499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.692423 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf" Oct 29 22:25:19.704934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.704913 4543 scope.go:122] "RemoveContainer" containerID="85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e" Oct 29 22:25:19.705249 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:19.705228 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e\": container with ID starting with 85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e not found: ID does not exist" containerID="85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e" Oct 29 22:25:19.705335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.705252 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e"} err="failed to get container status \"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e\": rpc error: code = NotFound desc = could not find container \"85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e\": container with ID starting with 85c9bbe7e191a6861d0328791943dce7275b8b31afaa8c8d86e074ad785d904e not found: ID does not exist" Oct 29 22:25:19.705335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.705269 4543 scope.go:122] "RemoveContainer" containerID="94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024" Oct 29 22:25:19.716159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.716142 4543 scope.go:122] "RemoveContainer" containerID="94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024" Oct 29 22:25:19.716431 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:19.716403 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024\": container with ID starting with 94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024 not found: ID does not exist" containerID="94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024" Oct 29 22:25:19.716487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.716438 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024"} err="failed to get container status \"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024\": rpc error: code = NotFound desc = could not find container \"94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024\": container with ID starting with 94e60a79461108358bcbfc4ce8c1cbf15df5209d919e5938b76122b0a358e024 not found: ID does not exist" Oct 29 22:25:19.727642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.727620 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:25:19.780098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.780062 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:19.786842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.786812 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-589b4c9d64-697cr"] Oct 29 22:25:19.827934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.827914 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:25:19.860797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:19.860773 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-5c2hf"] Oct 29 22:25:20.247116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.239454 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:20.247116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.239514 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:20.497109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.497047 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5139d749-7a35-4beb-ac83-76efbab179ce" path="/var/lib/kubelet/pods/5139d749-7a35-4beb-ac83-76efbab179ce/volumes" Oct 29 22:25:20.497993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.497933 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" path="/var/lib/kubelet/pods/bc224878-39b1-4043-9d1c-a2f6b607803a/volumes" Oct 29 22:25:20.708876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.707658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" event={"ID":"8b221315-7dd4-4019-839b-b5d27d83fc36","Type":"ContainerStarted","Data":"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05"} Oct 29 22:25:20.708876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.707710 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" event={"ID":"8b221315-7dd4-4019-839b-b5d27d83fc36","Type":"ContainerStarted","Data":"d4c3c3e985ce53a38d965231acdf4b9f0605b2e7a22308875f181adb5acd6148"} Oct 29 22:25:20.708876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.708803 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:20.755686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:20.755577 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" podStartSLOduration=2.755563498 podStartE2EDuration="2.755563498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:20.753445978 +0000 UTC m=+2582.495376652" watchObservedRunningTime="2026-10-29 22:25:20.755563498 +0000 UTC m=+2582.497494171" Oct 29 22:25:21.253558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:21.248444 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:21.253558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:21.251354 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:22.062278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.062240 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:25:22.063203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063184 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5139d749-7a35-4beb-ac83-76efbab179ce" containerName="payload-processing" Oct 29 22:25:22.063203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063201 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5139d749-7a35-4beb-ac83-76efbab179ce" containerName="payload-processing" Oct 29 22:25:22.063402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063240 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" Oct 29 22:25:22.063402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063245 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" Oct 29 22:25:22.063402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063388 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc224878-39b1-4043-9d1c-a2f6b607803a" containerName="istio-proxy" Oct 29 22:25:22.063596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.063405 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5139d749-7a35-4beb-ac83-76efbab179ce" containerName="payload-processing" Oct 29 22:25:22.068738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.068717 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.090449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.090421 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:25:22.095797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.095452 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:25:22.095797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.095534 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-31abe10e24c1\"" Oct 29 22:25:22.095797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.095695 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"kube-root-ca.crt\"" Oct 29 22:25:22.095797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.095771 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-2490cdadee51\"" Oct 29 22:25:22.096604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.096488 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-e2e--b7017604-dockercfg-gkpdl\"" Oct 29 22:25:22.096604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.096595 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"routing-overlay\"" Oct 29 22:25:22.103701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.103678 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"openshift-service-ca.crt\"" Oct 29 22:25:22.166692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.166649 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9p7\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.166914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.166896 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.167016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.166997 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.167153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.167132 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.236461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.236422 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dfq6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body= Oct 29 22:25:22.236643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.236491 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" Oct 29 22:25:22.268624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.268590 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9p7\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.268771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.268736 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.268870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.268807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.268945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.268901 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.269683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.269613 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.269683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.269686 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.271778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.271756 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.293504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.293485 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9p7\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.385909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.385849 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:22.712060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.712038 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:25:22.713370 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:25:22.713345 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f87e63_fb77_483d_9e4c_d55ef00e31e5.slice/crio-ef8e9030eac1b637e622307b0320fca7a4ad04705f2de2f256b2866a04ef1b94 WatchSource:0}: Error finding container ef8e9030eac1b637e622307b0320fca7a4ad04705f2de2f256b2866a04ef1b94: Status 404 returned error can't find the container with id ef8e9030eac1b637e622307b0320fca7a4ad04705f2de2f256b2866a04ef1b94 Oct 29 22:25:22.736522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.736498 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" event={"ID":"08f87e63-fb77-483d-9e4c-d55ef00e31e5","Type":"ContainerStarted","Data":"ef8e9030eac1b637e622307b0320fca7a4ad04705f2de2f256b2866a04ef1b94"} Oct 29 22:25:22.982471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.982435 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:25:22.989064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:22.989043 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.006333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.006311 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:25:23.012150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.012128 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-f0acd722a44c\"" Oct 29 22:25:23.081282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.081245 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.081707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.081298 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.081707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.081331 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.081707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.081402 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj2p\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.186579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.186544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.186852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.186826 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.187016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.186998 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.187252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.187234 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj2p\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.187995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.187969 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.188137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.187532 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.191625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.191537 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.217092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.217056 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj2p\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.308014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.307938 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:23.636265 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:25:23.636230 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669bb78c_ec79_497b_a576_988b357e03cd.slice/crio-880d48622d1b09e16aea2ce9711f4ca4c8a599875abafb2e03c486f500a02a17 WatchSource:0}: Error finding container 880d48622d1b09e16aea2ce9711f4ca4c8a599875abafb2e03c486f500a02a17: Status 404 returned error can't find the container with id 880d48622d1b09e16aea2ce9711f4ca4c8a599875abafb2e03c486f500a02a17 Oct 29 22:25:23.639327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.639303 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:25:23.726727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.726699 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:23.727257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.727233 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:23.728373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.728352 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:25:23.764303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.764222 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" event={"ID":"08f87e63-fb77-483d-9e4c-d55ef00e31e5","Type":"ContainerStarted","Data":"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8"} Oct 29 22:25:23.764502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.764306 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:23.766405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.766362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" event={"ID":"669bb78c-ec79-497b-a576-988b357e03cd","Type":"ContainerStarted","Data":"880d48622d1b09e16aea2ce9711f4ca4c8a599875abafb2e03c486f500a02a17"} Oct 29 22:25:23.875576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:23.875492 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" podStartSLOduration=1.875464738 podStartE2EDuration="1.875464738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:23.86819127 +0000 UTC m=+2585.610121944" watchObservedRunningTime="2026-10-29 22:25:23.875464738 +0000 UTC m=+2585.617395412" Oct 29 22:25:24.777397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:24.777357 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" event={"ID":"669bb78c-ec79-497b-a576-988b357e03cd","Type":"ContainerStarted","Data":"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d"} Oct 29 22:25:24.777854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:24.777425 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:24.820912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:24.820861 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" podStartSLOduration=2.8208471680000002 podStartE2EDuration="2.820847168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:24.819091903 +0000 UTC m=+2586.561022573" watchObservedRunningTime="2026-10-29 22:25:24.820847168 +0000 UTC m=+2586.562777841" Oct 29 22:25:26.524692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:26.524666 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:27.962835 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:27.962800 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:25:32.739360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:32.739331 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:25:34.780791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:34.780765 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:25:35.791527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:35.791501 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:25:35.896712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:35.896676 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:40.897982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:40.897938 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" podUID="c715da87-029e-4d30-a48b-383918d23e0d" containerName="payload-processing" containerID="cri-o://1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03" gracePeriod=25 Oct 29 22:25:41.196517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.196491 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:41.365013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.364984 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\") pod \"c715da87-029e-4d30-a48b-383918d23e0d\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " Oct 29 22:25:41.365210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.365157 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\") pod \"c715da87-029e-4d30-a48b-383918d23e0d\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " Oct 29 22:25:41.365314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.365220 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\") pod \"c715da87-029e-4d30-a48b-383918d23e0d\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " Oct 29 22:25:41.365314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.365269 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\") pod \"c715da87-029e-4d30-a48b-383918d23e0d\" (UID: \"c715da87-029e-4d30-a48b-383918d23e0d\") " Oct 29 22:25:41.365647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.365622 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume" pod "c715da87-029e-4d30-a48b-383918d23e0d" (UID: "c715da87-029e-4d30-a48b-383918d23e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:41.365755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.365703 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume" pod "c715da87-029e-4d30-a48b-383918d23e0d" (UID: "c715da87-029e-4d30-a48b-383918d23e0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:41.366169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.366149 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:41.366232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.366175 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c715da87-029e-4d30-a48b-383918d23e0d-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:41.367393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.367370 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials" pod "c715da87-029e-4d30-a48b-383918d23e0d" (UID: "c715da87-029e-4d30-a48b-383918d23e0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:41.367568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.367547 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4" pod "c715da87-029e-4d30-a48b-383918d23e0d" (UID: "c715da87-029e-4d30-a48b-383918d23e0d"). InnerVolumeSpecName "kube-api-access-sg9d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:41.468261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.468230 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:41.468261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:41.468258 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9d4\" (UniqueName: \"kubernetes.io/projected/c715da87-029e-4d30-a48b-383918d23e0d-kube-api-access-sg9d4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:42.037102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.035889 4543 generic.go:356] "Generic (PLEG): container finished" podID="c715da87-029e-4d30-a48b-383918d23e0d" containerID="1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03" exitCode=0 Oct 29 22:25:42.037102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.036361 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" event={"ID":"c715da87-029e-4d30-a48b-383918d23e0d","Type":"ContainerDied","Data":"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03"} Oct 29 22:25:42.037102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.036441 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" event={"ID":"c715da87-029e-4d30-a48b-383918d23e0d","Type":"ContainerDied","Data":"2dfdd1db3c4e4c31026b67a489a1d6f25672020a8a9c5a8a48a6f6278b1ac666"} Oct 29 22:25:42.037102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.036490 4543 scope.go:122] "RemoveContainer" containerID="1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03" Oct 29 22:25:42.037102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.036692 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp" Oct 29 22:25:42.157327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.157297 4543 scope.go:122] "RemoveContainer" containerID="1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03" Oct 29 22:25:42.158027 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:42.158001 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03\": container with ID starting with 1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03 not found: ID does not exist" containerID="1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03" Oct 29 22:25:42.158240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.158212 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03"} err="failed to get container status \"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03\": rpc error: code = NotFound desc = could not find container \"1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03\": container with ID starting with 1221fcff38a91121f8f48f261556ca253adee2ae146c05be0c7f3153968aeb03 not found: ID does not exist" Oct 29 22:25:42.265103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.261114 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:42.285451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.285421 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5474cw7wp"] Oct 29 22:25:42.712288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:42.712252 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c715da87-029e-4d30-a48b-383918d23e0d" path="/var/lib/kubelet/pods/c715da87-029e-4d30-a48b-383918d23e0d/volumes" Oct 29 22:25:43.206406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.206330 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7"] Oct 29 22:25:43.207398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.207372 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c715da87-029e-4d30-a48b-383918d23e0d" containerName="payload-processing" Oct 29 22:25:43.207398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.207393 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c715da87-029e-4d30-a48b-383918d23e0d" containerName="payload-processing" Oct 29 22:25:43.207570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.207524 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c715da87-029e-4d30-a48b-383918d23e0d" containerName="payload-processing" Oct 29 22:25:43.213159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.213141 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.242151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.242070 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:25:43.242474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.242450 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:25:43.246645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.246625 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7"] Oct 29 22:25:43.327697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.327659 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:25:43.327909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.327718 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:25:43.340415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.340392 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwpk\" (UniqueName: \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.340733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.340714 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.340865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.340845 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.443788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.443751 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwpk\" (UniqueName: \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.445193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.445165 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.445545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.445521 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.447366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.446969 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.463671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.462758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.470360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.470330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwpk\" (UniqueName: \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\") pod \"maas-api-revoke-keys-e2e-explicit-529e60-qhnv7\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:43.528679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:43.528648 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:44.114459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:44.109752 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7"] Oct 29 22:25:45.129559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:45.129463 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" event={"ID":"e383c994-116b-40dc-9ea6-d31ece8a9a75","Type":"ContainerStarted","Data":"e41c56c7bb9f34aa4529af31faa373fbbd9d7ab7817c9c8a724f11cb048cab8e"} Oct 29 22:25:45.130151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:45.130122 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" event={"ID":"e383c994-116b-40dc-9ea6-d31ece8a9a75","Type":"ContainerStarted","Data":"2a5a268c1f9ed1855525380ccdeef570d251bca1ae4154984c79db2e606c31f3"} Oct 29 22:25:45.183808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:45.183736 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" podStartSLOduration=2.183718527 podStartE2EDuration="2.183718527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:45.174738022 +0000 UTC m=+2606.916668697" watchObservedRunningTime="2026-10-29 22:25:45.183718527 +0000 UTC m=+2606.925649200" Oct 29 22:25:46.140931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:46.140900 4543 generic.go:356] "Generic (PLEG): container finished" podID="e383c994-116b-40dc-9ea6-d31ece8a9a75" containerID="e41c56c7bb9f34aa4529af31faa373fbbd9d7ab7817c9c8a724f11cb048cab8e" exitCode=0 Oct 29 22:25:46.141354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:46.140981 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" event={"ID":"e383c994-116b-40dc-9ea6-d31ece8a9a75","Type":"ContainerDied","Data":"e41c56c7bb9f34aa4529af31faa373fbbd9d7ab7817c9c8a724f11cb048cab8e"} Oct 29 22:25:47.332557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.332532 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:47.457711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.457630 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\") pod \"e383c994-116b-40dc-9ea6-d31ece8a9a75\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " Oct 29 22:25:47.457711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.457703 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\" (UniqueName: \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\") pod \"e383c994-116b-40dc-9ea6-d31ece8a9a75\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " Oct 29 22:25:47.457944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.457909 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\") pod \"e383c994-116b-40dc-9ea6-d31ece8a9a75\" (UID: \"e383c994-116b-40dc-9ea6-d31ece8a9a75\") " Oct 29 22:25:47.458438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.458284 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca" pod "e383c994-116b-40dc-9ea6-d31ece8a9a75" (UID: "e383c994-116b-40dc-9ea6-d31ece8a9a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:47.459054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.459031 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:47.460178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.460154 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca" pod "e383c994-116b-40dc-9ea6-d31ece8a9a75" (UID: "e383c994-116b-40dc-9ea6-d31ece8a9a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:25:47.460270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.460154 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk" pod "e383c994-116b-40dc-9ea6-d31ece8a9a75" (UID: "e383c994-116b-40dc-9ea6-d31ece8a9a75"). InnerVolumeSpecName "kube-api-access-rnwpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:47.560227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.560199 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwpk\" (UniqueName: \"kubernetes.io/projected/e383c994-116b-40dc-9ea6-d31ece8a9a75-kube-api-access-rnwpk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:47.560227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:47.560224 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e383c994-116b-40dc-9ea6-d31ece8a9a75-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:48.160361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:48.160319 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" event={"ID":"e383c994-116b-40dc-9ea6-d31ece8a9a75","Type":"ContainerDied","Data":"2a5a268c1f9ed1855525380ccdeef570d251bca1ae4154984c79db2e606c31f3"} Oct 29 22:25:48.160535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:48.160382 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7" Oct 29 22:25:48.160535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:48.160419 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5a268c1f9ed1855525380ccdeef570d251bca1ae4154984c79db2e606c31f3" Oct 29 22:25:53.727206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:53.727173 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:25:54.486460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:54.486429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:25:54.486637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:54.486604 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" containerID="cri-o://e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814" gracePeriod=30 Oct 29 22:25:57.962230 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:25:57.962192 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:25:59.213393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.213355 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" podUID="7138bb49-f6a3-46d2-9e84-788a4def2506" containerName="payload-pre-processing" containerID="cri-o://7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f" gracePeriod=25 Oct 29 22:25:59.580668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.580640 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:25:59.685365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.685335 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\" (UniqueName: \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\") pod \"7138bb49-f6a3-46d2-9e84-788a4def2506\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " Oct 29 22:25:59.685546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.685531 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\") pod \"7138bb49-f6a3-46d2-9e84-788a4def2506\" (UID: \"7138bb49-f6a3-46d2-9e84-788a4def2506\") " Oct 29 22:25:59.685958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.685930 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume" pod "7138bb49-f6a3-46d2-9e84-788a4def2506" (UID: "7138bb49-f6a3-46d2-9e84-788a4def2506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:59.686417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.686396 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138bb49-f6a3-46d2-9e84-788a4def2506-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.688167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.688136 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2" pod "7138bb49-f6a3-46d2-9e84-788a4def2506" (UID: "7138bb49-f6a3-46d2-9e84-788a4def2506"). InnerVolumeSpecName "kube-api-access-bnjk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.776237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.776213 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:25:59.787775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.787752 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjk2\" (UniqueName: \"kubernetes.io/projected/7138bb49-f6a3-46d2-9e84-788a4def2506-kube-api-access-bnjk2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.888864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.888831 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.888869 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.888957 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889046 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889166 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889183 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.889339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889231 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889286 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889345 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889373 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.889450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\") pod \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\" (UID: \"58d2ad4a-9d76-43f9-be12-2d14c6c21e53\") " Oct 29 22:25:59.889630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889463 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.889725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889688 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.889786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.889715 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:59.890568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.890540 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.890706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.890572 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.890706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.890588 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.890706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.890601 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.890706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.890615 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.891642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.891615 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:25:59.891897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.891871 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:59.891966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.891905 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.891966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.891937 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh" pod "58d2ad4a-9d76-43f9-be12-2d14c6c21e53" (UID: "58d2ad4a-9d76-43f9-be12-2d14c6c21e53"). InnerVolumeSpecName "kube-api-access-886jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.992456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.992425 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.992456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.992453 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.992456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.992462 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.992456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:25:59.992471 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886jh\" (UniqueName: \"kubernetes.io/projected/58d2ad4a-9d76-43f9-be12-2d14c6c21e53-kube-api-access-886jh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:00.269435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.269395 4543 generic.go:356] "Generic (PLEG): container finished" podID="7138bb49-f6a3-46d2-9e84-788a4def2506" containerID="7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f" exitCode=0 Oct 29 22:26:00.269820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.269443 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" Oct 29 22:26:00.269820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.269478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" event={"ID":"7138bb49-f6a3-46d2-9e84-788a4def2506","Type":"ContainerDied","Data":"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f"} Oct 29 22:26:00.269820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.269530 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7" event={"ID":"7138bb49-f6a3-46d2-9e84-788a4def2506","Type":"ContainerDied","Data":"743c7a9c64b18948d1e4a0c4d28cac37ec631af06608ead68faa1648cc67a58e"} Oct 29 22:26:00.269820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.269551 4543 scope.go:122] "RemoveContainer" containerID="7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f" Oct 29 22:26:00.271719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.271697 4543 generic.go:356] "Generic (PLEG): container finished" podID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerID="e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814" exitCode=0 Oct 29 22:26:00.271826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.271769 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" Oct 29 22:26:00.271826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.271776 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerDied","Data":"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814"} Oct 29 22:26:00.271826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.271822 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb" event={"ID":"58d2ad4a-9d76-43f9-be12-2d14c6c21e53","Type":"ContainerDied","Data":"6f917318d4e8427386aa0fe49c0c83d9d0b084d9ca7843e583ba6eb57cfda7a8"} Oct 29 22:26:00.282678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.282660 4543 scope.go:122] "RemoveContainer" containerID="7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f" Oct 29 22:26:00.282943 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:26:00.282922 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f\": container with ID starting with 7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f not found: ID does not exist" containerID="7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f" Oct 29 22:26:00.283020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.282950 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f"} err="failed to get container status \"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f\": rpc error: code = NotFound desc = could not find container \"7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f\": container with ID starting with 7bdc96d2b221d3395c2619b4a50369dfcac7cc2f6697bff164cc26a9fceb1e8f not found: ID does not exist" Oct 29 22:26:00.283020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.282967 4543 scope.go:122] "RemoveContainer" containerID="e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814" Oct 29 22:26:00.293494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.293479 4543 scope.go:122] "RemoveContainer" containerID="a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7" Oct 29 22:26:00.303725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.303705 4543 scope.go:122] "RemoveContainer" containerID="e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814" Oct 29 22:26:00.304024 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:26:00.304006 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814\": container with ID starting with e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814 not found: ID does not exist" containerID="e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814" Oct 29 22:26:00.304116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.304032 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814"} err="failed to get container status \"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814\": rpc error: code = NotFound desc = could not find container \"e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814\": container with ID starting with e075eaed79cf76cf3f7c9f7e2e367da493c7628b6ccadbff6f50b7d159e9e814 not found: ID does not exist" Oct 29 22:26:00.304116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.304049 4543 scope.go:122] "RemoveContainer" containerID="a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7" Oct 29 22:26:00.304356 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:26:00.304340 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7\": container with ID starting with a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7 not found: ID does not exist" containerID="a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7" Oct 29 22:26:00.304410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.304361 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7"} err="failed to get container status \"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7\": rpc error: code = NotFound desc = could not find container \"a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7\": container with ID starting with a38ecc48f472e0a4f8096ca2a6a47c63e8f8d0dd66befb01cf4d2e3f56d736b7 not found: ID does not exist" Oct 29 22:26:00.327561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.327538 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:26:00.339182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.339162 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-6tqdb"] Oct 29 22:26:00.368343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.368147 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:26:00.373495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.373472 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-529e60-8598b6cb59-lpzh7"] Oct 29 22:26:00.388969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.388944 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" path="/var/lib/kubelet/pods/58d2ad4a-9d76-43f9-be12-2d14c6c21e53/volumes" Oct 29 22:26:00.389428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:00.389412 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7138bb49-f6a3-46d2-9e84-788a4def2506" path="/var/lib/kubelet/pods/7138bb49-f6a3-46d2-9e84-788a4def2506/volumes" Oct 29 22:26:15.228970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:15.228926 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7"] Oct 29 22:26:15.271914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:15.271870 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-529e60-qhnv7"] Oct 29 22:26:16.390479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:16.390443 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e383c994-116b-40dc-9ea6-d31ece8a9a75" path="/var/lib/kubelet/pods/e383c994-116b-40dc-9ea6-d31ece8a9a75/volumes" Oct 29 22:26:23.206111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:23.206037 4543 scope.go:122] "RemoveContainer" containerID="96b9b95da6ffa807a718d088f0bf4c404eed5ece5a94d8b181406549d13fdad0" Oct 29 22:26:23.218045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:23.218028 4543 scope.go:122] "RemoveContainer" containerID="27fc498ce37dfdaa2a21dc861e68661ee3c91698b5e5e72a6d3fa5ee3cb3dd4a" Oct 29 22:26:23.248258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:23.247536 4543 scope.go:122] "RemoveContainer" containerID="8ee52fe0da518f1ae556f4b0ea2bb1fc62310483b085c942225ae9fb5774f307" Oct 29 22:26:23.263524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:23.263499 4543 scope.go:122] "RemoveContainer" containerID="dbc8c48f9fcdc1892c9d0583fcb1539898afa2e4aa734b4237163ce5de41baab" Oct 29 22:26:27.961492 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:26:27.961455 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:26:31.766357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.766278 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:26:31.767344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767321 4543 cpu_manager.go:414] "containerMap: removing container" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.767344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767341 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767361 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7138bb49-f6a3-46d2-9e84-788a4def2506" containerName="payload-pre-processing" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767366 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7138bb49-f6a3-46d2-9e84-788a4def2506" containerName="payload-pre-processing" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767392 4543 cpu_manager.go:414] "containerMap: removing container" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767396 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767416 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e383c994-116b-40dc-9ea6-d31ece8a9a75" containerName="revoke-keys" Oct 29 22:26:31.767506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767421 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e383c994-116b-40dc-9ea6-d31ece8a9a75" containerName="revoke-keys" Oct 29 22:26:31.767707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767584 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7138bb49-f6a3-46d2-9e84-788a4def2506" containerName="payload-pre-processing" Oct 29 22:26:31.767707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767601 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.767707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.767610 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e383c994-116b-40dc-9ea6-d31ece8a9a75" containerName="revoke-keys" Oct 29 22:26:31.774042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.774016 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:31.800817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.800791 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:26:31.805762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.805735 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-749f72\"" Oct 29 22:26:31.805906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.805857 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-749f72-dockercfg-jlrc2\"" Oct 29 22:26:31.842488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.842460 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:26:31.844261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.844239 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d2ad4a-9d76-43f9-be12-2d14c6c21e53" containerName="istio-proxy" Oct 29 22:26:31.849062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.849034 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:31.849354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.849321 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5lh\" (UniqueName: \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:31.849498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.849471 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:31.870846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:31.870819 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:26:32.030219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.011856 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5lh\" (UniqueName: \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.030219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.011939 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmxb\" (UniqueName: \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.030219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.012259 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.030219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.012379 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.030219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.013315 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.131381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.123822 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5lh\" (UniqueName: \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\") pod \"payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.131381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.123935 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.131381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.124459 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmxb\" (UniqueName: \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.131381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.124841 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.131381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.126473 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.226220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.216383 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmxb\" (UniqueName: \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\") pod \"payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.480844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.480810 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:32.509327 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:26:32.509296 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718bbebf_8ef8_4311_a934_7b28acfc5cfd.slice/crio-794b4b26c343c8e78db2b428fc141a8994a45715bc14fe882ab77f07648909bf WatchSource:0}: Error finding container 794b4b26c343c8e78db2b428fc141a8994a45715bc14fe882ab77f07648909bf: Status 404 returned error can't find the container with id 794b4b26c343c8e78db2b428fc141a8994a45715bc14fe882ab77f07648909bf Oct 29 22:26:32.511862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.511839 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:26:32.672672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.672636 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" event={"ID":"718bbebf-8ef8-4311-a934-7b28acfc5cfd","Type":"ContainerStarted","Data":"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef"} Oct 29 22:26:32.672830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.672692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" event={"ID":"718bbebf-8ef8-4311-a934-7b28acfc5cfd","Type":"ContainerStarted","Data":"794b4b26c343c8e78db2b428fc141a8994a45715bc14fe882ab77f07648909bf"} Oct 29 22:26:32.672830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.672722 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:32.723644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.723604 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" podStartSLOduration=1.723589775 podStartE2EDuration="1.723589775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:32.722757926 +0000 UTC m=+2654.464688612" watchObservedRunningTime="2026-10-29 22:26:32.723589775 +0000 UTC m=+2654.465520448" Oct 29 22:26:32.804871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:32.804843 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:26:32.805389 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:26:32.805126 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5924a540_1317_4e5a_9251_9ced453c35be.slice/crio-c77a068d236adc0749f3f88320cd187f15d4522bfc9a31e74dee4d7972c69f6b WatchSource:0}: Error finding container c77a068d236adc0749f3f88320cd187f15d4522bfc9a31e74dee4d7972c69f6b: Status 404 returned error can't find the container with id c77a068d236adc0749f3f88320cd187f15d4522bfc9a31e74dee4d7972c69f6b Oct 29 22:26:33.683394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:33.683363 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" event={"ID":"5924a540-1317-4e5a-9251-9ced453c35be","Type":"ContainerStarted","Data":"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846"} Oct 29 22:26:33.683629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:33.683415 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" event={"ID":"5924a540-1317-4e5a-9251-9ced453c35be","Type":"ContainerStarted","Data":"c77a068d236adc0749f3f88320cd187f15d4522bfc9a31e74dee4d7972c69f6b"} Oct 29 22:26:33.683629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:33.683442 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:33.725563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:33.725513 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" podStartSLOduration=2.725495741 podStartE2EDuration="2.725495741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:33.724783987 +0000 UTC m=+2655.466714661" watchObservedRunningTime="2026-10-29 22:26:33.725495741 +0000 UTC m=+2655.467426415" Oct 29 22:26:39.058761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.058728 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v"] Oct 29 22:26:39.064258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.064234 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.073234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.073203 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v"] Oct 29 22:26:39.082676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.082542 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:26:39.093597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.093571 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:26:39.093738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.093638 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:26:39.093801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.093744 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:26:39.096188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.096098 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.096323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.096297 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.096485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.096463 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5h7\" (UniqueName: \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.198279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.198245 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.198423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.198330 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.198423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.198403 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5h7\" (UniqueName: \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.199019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.198994 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.200896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.200875 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.223022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.222998 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5h7\" (UniqueName: \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\") pod \"maas-api-revoke-sub-a8af1539-78f6v\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.385758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.385679 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:39.727049 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:26:39.727017 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a34efe0_9615_4fd4_940f_c70aff824c78.slice/crio-43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1 WatchSource:0}: Error finding container 43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1: Status 404 returned error can't find the container with id 43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1 Oct 29 22:26:39.727927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.727904 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v"] Oct 29 22:26:39.875330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:39.875297 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" event={"ID":"9a34efe0-9615-4fd4-940f-c70aff824c78","Type":"ContainerStarted","Data":"43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1"} Oct 29 22:26:40.895491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:40.895441 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" event={"ID":"9a34efe0-9615-4fd4-940f-c70aff824c78","Type":"ContainerStarted","Data":"864d38fae0fd4a0182896e225783c2b0ea020020564bc9bf3afbcc0eb0dae8f1"} Oct 29 22:26:40.960054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:40.959996 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" podStartSLOduration=1.9599772949999998 podStartE2EDuration="1.959977295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:40.943471183 +0000 UTC m=+2662.685401856" watchObservedRunningTime="2026-10-29 22:26:40.959977295 +0000 UTC m=+2662.701907969" Oct 29 22:26:41.905615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:41.905584 4543 generic.go:356] "Generic (PLEG): container finished" podID="9a34efe0-9615-4fd4-940f-c70aff824c78" containerID="864d38fae0fd4a0182896e225783c2b0ea020020564bc9bf3afbcc0eb0dae8f1" exitCode=0 Oct 29 22:26:41.906145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:41.905680 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" event={"ID":"9a34efe0-9615-4fd4-940f-c70aff824c78","Type":"ContainerDied","Data":"864d38fae0fd4a0182896e225783c2b0ea020020564bc9bf3afbcc0eb0dae8f1"} Oct 29 22:26:43.246892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.246491 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:43.412989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.412950 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\") pod \"9a34efe0-9615-4fd4-940f-c70aff824c78\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " Oct 29 22:26:43.413209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.413013 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\" (UniqueName: \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\") pod \"9a34efe0-9615-4fd4-940f-c70aff824c78\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " Oct 29 22:26:43.413209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.413182 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\") pod \"9a34efe0-9615-4fd4-940f-c70aff824c78\" (UID: \"9a34efe0-9615-4fd4-940f-c70aff824c78\") " Oct 29 22:26:43.413597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.413565 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca" pod "9a34efe0-9615-4fd4-940f-c70aff824c78" (UID: "9a34efe0-9615-4fd4-940f-c70aff824c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:26:43.414422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.414400 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:43.415410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.415388 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca" pod "9a34efe0-9615-4fd4-940f-c70aff824c78" (UID: "9a34efe0-9615-4fd4-940f-c70aff824c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:43.415619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.415596 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7" pod "9a34efe0-9615-4fd4-940f-c70aff824c78" (UID: "9a34efe0-9615-4fd4-940f-c70aff824c78"). InnerVolumeSpecName "kube-api-access-wl5h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:43.516184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.516111 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a34efe0-9615-4fd4-940f-c70aff824c78-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:43.516184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.516151 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5h7\" (UniqueName: \"kubernetes.io/projected/9a34efe0-9615-4fd4-940f-c70aff824c78-kube-api-access-wl5h7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:43.686507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.686485 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:26:43.930165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.930066 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" Oct 29 22:26:43.930165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.930106 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v" event={"ID":"9a34efe0-9615-4fd4-940f-c70aff824c78","Type":"ContainerDied","Data":"43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1"} Oct 29 22:26:43.930165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:43.930150 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43857c9de4d800d9e88aacea06da8ddd5a0a869126295714a258b1fbd54797f1" Oct 29 22:26:44.036565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:44.036532 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt"] Oct 29 22:26:44.044390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:44.044347 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b8fea0-9w9wt"] Oct 29 22:26:44.393699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:44.393661 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e2bab6-4000-412d-8215-331ecb6d99fa" path="/var/lib/kubelet/pods/20e2bab6-4000-412d-8215-331ecb6d99fa/volumes" Oct 29 22:26:44.721575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:44.719827 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:26:48.011516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.011484 4543 generic.go:356] "Generic (PLEG): container finished" podID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerID="855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d" exitCode=0 Oct 29 22:26:48.012137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.011562 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerDied","Data":"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d"} Oct 29 22:26:48.012137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.012032 4543 scope.go:122] "RemoveContainer" containerID="855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d" Oct 29 22:26:48.013963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.013932 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:48.014069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.014010 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:48.014069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:48.014047 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:49.025940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:49.025907 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerStarted","Data":"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5"} Oct 29 22:26:49.627112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:49.625551 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:26:49.627112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:49.625679 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:49.627112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:49.625735 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:50.620528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:50.620498 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:50.620922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:50.620560 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:51.622902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.622704 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:51.622902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.622780 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:51.965421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.965148 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv"] Oct 29 22:26:51.967222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.966820 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9a34efe0-9615-4fd4-940f-c70aff824c78" containerName="revoke-keys" Oct 29 22:26:51.967222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.966846 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a34efe0-9615-4fd4-940f-c70aff824c78" containerName="revoke-keys" Oct 29 22:26:51.967222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.967150 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a34efe0-9615-4fd4-940f-c70aff824c78" containerName="revoke-keys" Oct 29 22:26:51.971448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:51.971426 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.002295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.002263 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv"] Oct 29 22:26:52.011497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.008292 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.011497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.008421 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:26:52.011497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.008486 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.011497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.008510 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:26:52.011497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.008797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf58\" (UniqueName: \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.092120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.052462 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:26:52.092120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.052547 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:26:52.117187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.117151 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf58\" (UniqueName: \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.123997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.123935 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.124510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.124434 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.125875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.125745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.127353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.126394 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.152892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.152809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf58\" (UniqueName: \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\") pod \"maas-api-revoke-sub-c20815ee-7n9zv\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.388526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.388135 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:52.620155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.620124 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:52.620294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.620187 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:52.824329 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:26:52.824295 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91586ca4_47ae_4f76_938c_4e4d39f99a83.slice/crio-ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1 WatchSource:0}: Error finding container ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1: Status 404 returned error can't find the container with id ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1 Oct 29 22:26:52.825361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:52.825281 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv"] Oct 29 22:26:53.183490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:53.183413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" event={"ID":"91586ca4-47ae-4f76-938c-4e4d39f99a83","Type":"ContainerStarted","Data":"ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1"} Oct 29 22:26:53.620420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:53.620390 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:53.620588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:53.620449 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:54.059059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.059019 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv"] Oct 29 22:26:54.067399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.067378 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed43f451-md8fv"] Oct 29 22:26:54.273968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.273640 4543 generic.go:356] "Generic (PLEG): container finished" podID="91586ca4-47ae-4f76-938c-4e4d39f99a83" containerID="4af5b83af1a49f38beba6b19373acd41cec88ce489c8f6a11e4d0b6c6ce37d67" exitCode=0 Oct 29 22:26:54.273968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.273722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" event={"ID":"91586ca4-47ae-4f76-938c-4e4d39f99a83","Type":"ContainerDied","Data":"4af5b83af1a49f38beba6b19373acd41cec88ce489c8f6a11e4d0b6c6ce37d67"} Oct 29 22:26:54.389611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.389533 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee42192e-59ef-41ba-b917-268e18a8c2fa" path="/var/lib/kubelet/pods/ee42192e-59ef-41ba-b917-268e18a8c2fa/volumes" Oct 29 22:26:54.619966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.619929 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" start-of-body= Oct 29 22:26:54.620180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:54.619987 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.87:15021/healthz/ready\": dial tcp 10.133.1.87:15021: connect: connection refused" Oct 29 22:26:55.469341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.469317 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:55.540316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.540247 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\") pod \"91586ca4-47ae-4f76-938c-4e4d39f99a83\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " Oct 29 22:26:55.540451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.540419 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\") pod \"91586ca4-47ae-4f76-938c-4e4d39f99a83\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " Oct 29 22:26:55.540565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.540547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\" (UniqueName: \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\") pod \"91586ca4-47ae-4f76-938c-4e4d39f99a83\" (UID: \"91586ca4-47ae-4f76-938c-4e4d39f99a83\") " Oct 29 22:26:55.540809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.540783 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca" pod "91586ca4-47ae-4f76-938c-4e4d39f99a83" (UID: "91586ca4-47ae-4f76-938c-4e4d39f99a83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:26:55.541372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.541345 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:55.542767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.542739 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca" pod "91586ca4-47ae-4f76-938c-4e4d39f99a83" (UID: "91586ca4-47ae-4f76-938c-4e4d39f99a83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:26:55.543183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.543157 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58" pod "91586ca4-47ae-4f76-938c-4e4d39f99a83" (UID: "91586ca4-47ae-4f76-938c-4e4d39f99a83"). InnerVolumeSpecName "kube-api-access-mvf58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:26:55.625736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.625709 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:26:55.626210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.626187 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:26:55.626813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.626795 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:26:55.642547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.642523 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91586ca4-47ae-4f76-938c-4e4d39f99a83-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:55.642547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:55.642545 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvf58\" (UniqueName: \"kubernetes.io/projected/91586ca4-47ae-4f76-938c-4e4d39f99a83-kube-api-access-mvf58\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:26:56.293503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:56.293473 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" Oct 29 22:26:56.293671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:56.293506 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv" event={"ID":"91586ca4-47ae-4f76-938c-4e4d39f99a83","Type":"ContainerDied","Data":"ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1"} Oct 29 22:26:56.293671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:56.293584 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec07f9fa49ae60eb1583ae6f5ac779be41a5622f9bdb0c91362ad5cd53c480d1" Oct 29 22:26:57.578518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.578483 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:26:57.579675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.579650 4543 cpu_manager.go:414] "containerMap: removing container" podUID="91586ca4-47ae-4f76-938c-4e4d39f99a83" containerName="revoke-keys" Oct 29 22:26:57.579675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.579673 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91586ca4-47ae-4f76-938c-4e4d39f99a83" containerName="revoke-keys" Oct 29 22:26:57.579974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.579958 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="91586ca4-47ae-4f76-938c-4e4d39f99a83" containerName="revoke-keys" Oct 29 22:26:57.584226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.584207 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.614097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.614056 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-07e166-openshift-default-dockercfg-nf577\"" Oct 29 22:26:57.622834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.622808 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:26:57.689736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.689709 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spt7s\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.689858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.689837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.689926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.689867 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.689995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.689947 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.690054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.690024 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.690167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.690151 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.690240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.690182 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.690313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.690301 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.690362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.690353 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.791994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.791956 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spt7s\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792146 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792289 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792383 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792486 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792588 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792626 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792676 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792755 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792774 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.792967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.792934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.793397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.793370 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.795241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.795219 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.795487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.795471 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.816805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.816774 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.816925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.816907 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spt7s\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.899768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:57.899709 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:57.969926 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:26:57.969867 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:26:58.252131 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:26:58.252101 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb8c834_1ac4_41a2_a059_9df980779ae5.slice/crio-3a6166b3287e73b14df4e3b792383508776c48e5ad2b3251232158a8fab71460 WatchSource:0}: Error finding container 3a6166b3287e73b14df4e3b792383508776c48e5ad2b3251232158a8fab71460: Status 404 returned error can't find the container with id 3a6166b3287e73b14df4e3b792383508776c48e5ad2b3251232158a8fab71460 Oct 29 22:26:58.252533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:58.252514 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:26:58.255020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:58.254984 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:58.255155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:58.255068 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:58.255155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:58.255125 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:26:58.315749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:58.315717 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" event={"ID":"8fb8c834-1ac4-41a2-a059-9df980779ae5","Type":"ContainerStarted","Data":"3a6166b3287e73b14df4e3b792383508776c48e5ad2b3251232158a8fab71460"} Oct 29 22:26:59.326785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:59.326746 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" event={"ID":"8fb8c834-1ac4-41a2-a059-9df980779ae5","Type":"ContainerStarted","Data":"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b"} Oct 29 22:26:59.406631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:59.406575 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podStartSLOduration=2.406561357 podStartE2EDuration="2.406561357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:59.38989692 +0000 UTC m=+2681.131827593" watchObservedRunningTime="2026-10-29 22:26:59.406561357 +0000 UTC m=+2681.148492030" Oct 29 22:26:59.900447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:59.900415 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:26:59.902433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:59.902406 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 22:26:59.902572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:26:59.902455 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 22:27:00.900300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:00.900263 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 22:27:00.900707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:00.900317 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 22:27:01.900041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:01.900003 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 22:27:01.900281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:01.900062 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 22:27:02.900354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:02.900319 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 22:27:02.900800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:02.900376 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 22:27:03.899994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:03.899964 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body= Oct 29 22:27:03.900176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:03.900025 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" Oct 29 22:27:04.306234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.306195 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72"] Oct 29 22:27:04.325158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.325123 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72"] Oct 29 22:27:04.325346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.325263 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.352511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.352479 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:27:04.363843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.363819 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:27:04.364015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.363889 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:27:04.364015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.363890 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:27:04.409737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.409711 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.410026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.410007 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x2k\" (UniqueName: \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.410233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.410213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.511237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.511205 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x2k\" (UniqueName: \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.511382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.511325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.511487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.511468 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.511997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.511972 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.513956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.513937 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.553890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.553864 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x2k\" (UniqueName: \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\") pod \"maas-api-revoke-sub-54d168ad-h9v72\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.668264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.668180 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:04.991353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:04.991324 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72"] Oct 29 22:27:04.991762 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:27:04.991730 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb0868a_c24a_4380_b835_a9a36f40be7b.slice/crio-1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb WatchSource:0}: Error finding container 1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb: Status 404 returned error can't find the container with id 1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb Oct 29 22:27:05.094130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.094102 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:27:05.095686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.095654 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:27:05.096583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.096551 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:27:05.514929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.514836 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" event={"ID":"8fb0868a-c24a-4380-b835-a9a36f40be7b","Type":"ContainerStarted","Data":"1b5eb5943163102f914910e84cc385ff4aaa522c89eb67035f9c731a0271efab"} Oct 29 22:27:05.514929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.514893 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" event={"ID":"8fb0868a-c24a-4380-b835-a9a36f40be7b","Type":"ContainerStarted","Data":"1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb"} Oct 29 22:27:05.559381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:05.559330 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" podStartSLOduration=2.559317022 podStartE2EDuration="2.559317022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:05.555544071 +0000 UTC m=+2687.297474743" watchObservedRunningTime="2026-10-29 22:27:05.559317022 +0000 UTC m=+2687.301247692" Oct 29 22:27:06.524818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:06.524786 4543 generic.go:356] "Generic (PLEG): container finished" podID="8fb0868a-c24a-4380-b835-a9a36f40be7b" containerID="1b5eb5943163102f914910e84cc385ff4aaa522c89eb67035f9c731a0271efab" exitCode=0 Oct 29 22:27:06.525235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:06.524866 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" event={"ID":"8fb0868a-c24a-4380-b835-a9a36f40be7b","Type":"ContainerDied","Data":"1b5eb5943163102f914910e84cc385ff4aaa522c89eb67035f9c731a0271efab"} Oct 29 22:27:07.731373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.731349 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:07.857556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.857485 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\") pod \"8fb0868a-c24a-4380-b835-a9a36f40be7b\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " Oct 29 22:27:07.857556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.857529 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\") pod \"8fb0868a-c24a-4380-b835-a9a36f40be7b\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " Oct 29 22:27:07.857900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.857874 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\" (UniqueName: \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\") pod \"8fb0868a-c24a-4380-b835-a9a36f40be7b\" (UID: \"8fb0868a-c24a-4380-b835-a9a36f40be7b\") " Oct 29 22:27:07.858025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.857927 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca" pod "8fb0868a-c24a-4380-b835-a9a36f40be7b" (UID: "8fb0868a-c24a-4380-b835-a9a36f40be7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:07.858471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.858448 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:07.859768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.859742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca" pod "8fb0868a-c24a-4380-b835-a9a36f40be7b" (UID: "8fb0868a-c24a-4380-b835-a9a36f40be7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:27:07.860151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.860129 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k" pod "8fb0868a-c24a-4380-b835-a9a36f40be7b" (UID: "8fb0868a-c24a-4380-b835-a9a36f40be7b"). InnerVolumeSpecName "kube-api-access-q7x2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:07.960300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.960275 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7x2k\" (UniqueName: \"kubernetes.io/projected/8fb0868a-c24a-4380-b835-a9a36f40be7b-kube-api-access-q7x2k\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:07.960300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:07.960300 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb0868a-c24a-4380-b835-a9a36f40be7b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:08.546381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:08.546349 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" Oct 29 22:27:08.546381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:08.546356 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72" event={"ID":"8fb0868a-c24a-4380-b835-a9a36f40be7b","Type":"ContainerDied","Data":"1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb"} Oct 29 22:27:08.546594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:08.546400 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aefd620ee880e036e47a02a2b135a858b91e0c49a679f0798f37f8beebe2feb" Oct 29 22:27:13.045463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:13.045429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f"] Oct 29 22:27:13.056529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:13.056501 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a3d33e-zg74f"] Oct 29 22:27:14.388737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:14.388702 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98443e6c-f276-43a7-b60d-eaff0bc36537" path="/var/lib/kubelet/pods/98443e6c-f276-43a7-b60d-eaff0bc36537/volumes" Oct 29 22:27:15.041248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:15.041216 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk"] Oct 29 22:27:15.052526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:15.052496 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b8305d-6hmjk"] Oct 29 22:27:15.144767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:15.144738 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:27:16.389393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:16.389357 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c000437-c23b-4f26-9ca0-2307174db1ff" path="/var/lib/kubelet/pods/6c000437-c23b-4f26-9ca0-2307174db1ff/volumes" Oct 29 22:27:20.145626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.145571 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" podUID="62bbbd28-50cd-4598-bc36-47956400ec28" containerName="payload-processing" containerID="cri-o://82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1" gracePeriod=25 Oct 29 22:27:20.458606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.458583 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:27:20.609091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609055 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\") pod \"62bbbd28-50cd-4598-bc36-47956400ec28\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " Oct 29 22:27:20.609346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609331 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\") pod \"62bbbd28-50cd-4598-bc36-47956400ec28\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " Oct 29 22:27:20.609407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609388 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\") pod \"62bbbd28-50cd-4598-bc36-47956400ec28\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " Oct 29 22:27:20.609540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609437 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\") pod \"62bbbd28-50cd-4598-bc36-47956400ec28\" (UID: \"62bbbd28-50cd-4598-bc36-47956400ec28\") " Oct 29 22:27:20.609782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609757 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume" pod "62bbbd28-50cd-4598-bc36-47956400ec28" (UID: "62bbbd28-50cd-4598-bc36-47956400ec28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:20.609846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.609784 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume" pod "62bbbd28-50cd-4598-bc36-47956400ec28" (UID: "62bbbd28-50cd-4598-bc36-47956400ec28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:20.610160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.610143 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.610221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.610162 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62bbbd28-50cd-4598-bc36-47956400ec28-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.611297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.611271 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials" pod "62bbbd28-50cd-4598-bc36-47956400ec28" (UID: "62bbbd28-50cd-4598-bc36-47956400ec28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:20.611735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.611717 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr" pod "62bbbd28-50cd-4598-bc36-47956400ec28" (UID: "62bbbd28-50cd-4598-bc36-47956400ec28"). InnerVolumeSpecName "kube-api-access-mgwdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:20.711482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.711460 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwdr\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-kube-api-access-mgwdr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.711595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.711485 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62bbbd28-50cd-4598-bc36-47956400ec28-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:20.724000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.723976 4543 generic.go:356] "Generic (PLEG): container finished" podID="62bbbd28-50cd-4598-bc36-47956400ec28" containerID="82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1" exitCode=0 Oct 29 22:27:20.724113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.724028 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" Oct 29 22:27:20.724113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.724062 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" event={"ID":"62bbbd28-50cd-4598-bc36-47956400ec28","Type":"ContainerDied","Data":"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1"} Oct 29 22:27:20.724217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.724131 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q" event={"ID":"62bbbd28-50cd-4598-bc36-47956400ec28","Type":"ContainerDied","Data":"209f166c97aa8404085b7ed8d84ea9cae992c05e1d0a0bd781577bc130478807"} Oct 29 22:27:20.724217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.724154 4543 scope.go:122] "RemoveContainer" containerID="82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1" Oct 29 22:27:20.736170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.736151 4543 scope.go:122] "RemoveContainer" containerID="82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1" Oct 29 22:27:20.736439 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:20.736416 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1\": container with ID starting with 82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1 not found: ID does not exist" containerID="82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1" Oct 29 22:27:20.736519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.736449 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1"} err="failed to get container status \"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1\": rpc error: code = NotFound desc = could not find container \"82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1\": container with ID starting with 82e322f6cfd8533a4a59b4a28ad10de34b305f95266e0f231d176d745fcf53e1 not found: ID does not exist" Oct 29 22:27:20.789230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.789197 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:27:20.810515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:20.810486 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-07e166/payload-processing-external-model-e2e-extmod-b-07e166-6764ndf5q"] Oct 29 22:27:22.390108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:22.390045 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bbbd28-50cd-4598-bc36-47956400ec28" path="/var/lib/kubelet/pods/62bbbd28-50cd-4598-bc36-47956400ec28/volumes" Oct 29 22:27:25.040917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:25.040887 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl"] Oct 29 22:27:25.052391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:25.052366 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38504de1-7xccl"] Oct 29 22:27:26.390052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:26.390014 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3872383-adf2-4a9a-8dab-b7a685ae60ee" path="/var/lib/kubelet/pods/d3872383-adf2-4a9a-8dab-b7a685ae60ee/volumes" Oct 29 22:27:27.962522 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:27.962486 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:27:34.366296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.366264 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:34.367419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367399 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8fb0868a-c24a-4380-b835-a9a36f40be7b" containerName="revoke-keys" Oct 29 22:27:34.367419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367419 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb0868a-c24a-4380-b835-a9a36f40be7b" containerName="revoke-keys" Oct 29 22:27:34.367571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367456 4543 cpu_manager.go:414] "containerMap: removing container" podUID="62bbbd28-50cd-4598-bc36-47956400ec28" containerName="payload-processing" Oct 29 22:27:34.367571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367461 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bbbd28-50cd-4598-bc36-47956400ec28" containerName="payload-processing" Oct 29 22:27:34.367645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367613 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb0868a-c24a-4380-b835-a9a36f40be7b" containerName="revoke-keys" Oct 29 22:27:34.367645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.367628 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bbbd28-50cd-4598-bc36-47956400ec28" containerName="payload-processing" Oct 29 22:27:34.373355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.373337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.402113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.402068 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408221 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"openshift-service-ca.crt\"" Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408231 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"agc-cred-010cbd9a21cf\"" Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408233 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0c3d9a\"" Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408289 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"kube-root-ca.crt\"" Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408327 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"routing-overlay\"" Oct 29 22:27:34.409090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.408360 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"payload-processing-external-model-e2e--95fdccdd-dockercfg-542wd\"" Oct 29 22:27:34.460294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.460268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.460432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.460415 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zs8\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.460523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.460508 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.460760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.460736 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.584131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.576482 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.587937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.584672 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zs8\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.587937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.584804 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.587937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.584957 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.606099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.584458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.606099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.590205 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.662923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.642182 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zs8\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.662923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.644950 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:34.698101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:34.697197 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:35.047616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.047586 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:27:35.053121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.053067 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.069176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.069156 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0c3d9a\"/\"agc-cred-d196180f917d\"" Oct 29 22:27:35.073669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.073647 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:27:35.100022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.099993 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.100203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.100036 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.100203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.100124 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.100358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.100315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vtxq\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.167066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.167039 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:35.167310 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:27:35.167288 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117a96c6_df8d_4db7_bfb3_23177f99e509.slice/crio-2b5c8269f15a151aeb453058c235281505ebc3564918c6de930656c77c90776e WatchSource:0}: Error finding container 2b5c8269f15a151aeb453058c235281505ebc3564918c6de930656c77c90776e: Status 404 returned error can't find the container with id 2b5c8269f15a151aeb453058c235281505ebc3564918c6de930656c77c90776e Oct 29 22:27:35.201345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.201323 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.201444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.201357 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.201444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.201394 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.201583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.201566 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vtxq\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.202114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.202035 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.202293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.202160 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.204490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.204464 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.233386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.233365 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vtxq\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\") pod \"payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.374135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.374053 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:35.688779 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:27:35.688753 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf952f7b4_61c2_46c2_9d52_d40a0b889ea1.slice/crio-46fe70859b1b8b448e6c59c7094ff5badaef6444c86c34003d485123fbd1d0f5 WatchSource:0}: Error finding container 46fe70859b1b8b448e6c59c7094ff5badaef6444c86c34003d485123fbd1d0f5: Status 404 returned error can't find the container with id 46fe70859b1b8b448e6c59c7094ff5badaef6444c86c34003d485123fbd1d0f5 Oct 29 22:27:35.689061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:35.689042 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:27:36.058408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.058377 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" event={"ID":"f952f7b4-61c2-46c2-9d52-d40a0b889ea1","Type":"ContainerStarted","Data":"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c"} Oct 29 22:27:36.058646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.058440 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" event={"ID":"f952f7b4-61c2-46c2-9d52-d40a0b889ea1","Type":"ContainerStarted","Data":"46fe70859b1b8b448e6c59c7094ff5badaef6444c86c34003d485123fbd1d0f5"} Oct 29 22:27:36.058646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.058470 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:36.060413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.060386 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" event={"ID":"117a96c6-df8d-4db7-bfb3-23177f99e509","Type":"ContainerStarted","Data":"8aa390fd0cd5755e97a610eedb193bedbcec5d4bf8944192f633e2ad2696f350"} Oct 29 22:27:36.060551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.060433 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:36.060551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.060452 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" event={"ID":"117a96c6-df8d-4db7-bfb3-23177f99e509","Type":"ContainerStarted","Data":"2b5c8269f15a151aeb453058c235281505ebc3564918c6de930656c77c90776e"} Oct 29 22:27:36.103699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.103650 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" podStartSLOduration=1.103637167 podStartE2EDuration="1.103637167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:36.100303172 +0000 UTC m=+2717.842233848" watchObservedRunningTime="2026-10-29 22:27:36.103637167 +0000 UTC m=+2717.845567839" Oct 29 22:27:36.145296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:36.145254 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" podStartSLOduration=2.145242453 podStartE2EDuration="2.145242453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:36.142288184 +0000 UTC m=+2717.884218858" watchObservedRunningTime="2026-10-29 22:27:36.145242453 +0000 UTC m=+2717.887173168" Oct 29 22:27:47.073193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.073164 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:47.073789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.073241 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:27:47.209831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.209794 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:47.972496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.972458 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl"] Oct 29 22:27:47.977684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.977658 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:47.996304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:47.996276 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl"] Oct 29 22:27:48.001639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.001617 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:27:48.001766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.001656 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:27:48.001766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.001664 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:27:48.005293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.005275 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:27:48.160477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.160446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltzh\" (UniqueName: \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.160901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.160673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.160901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.160782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.262035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.261965 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltzh\" (UniqueName: \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.262194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.262121 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.262194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.262182 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.262916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.262891 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.264688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.264671 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.319101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.317879 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltzh\" (UniqueName: \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\") pod \"maas-api-revoke-sub-1ecfee64-pw2nl\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:48.597593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:48.597516 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:49.215439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:49.204626 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl"] Oct 29 22:27:49.385857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:49.383171 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" event={"ID":"dad8b200-625c-4965-9290-287e6d25be96","Type":"ContainerStarted","Data":"fa747fa24af4e79e34432bc8d5a2227a2d68edc91c9ad27397ab6461b140590e"} Oct 29 22:27:50.394306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:50.394271 4543 generic.go:356] "Generic (PLEG): container finished" podID="dad8b200-625c-4965-9290-287e6d25be96" containerID="9b9bb27c033cec7f77d293cf84dcdde5973bd0c57823d2ba4821d81a47eb722d" exitCode=0 Oct 29 22:27:50.394741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:50.394483 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" event={"ID":"dad8b200-625c-4965-9290-287e6d25be96","Type":"ContainerDied","Data":"9b9bb27c033cec7f77d293cf84dcdde5973bd0c57823d2ba4821d81a47eb722d"} Oct 29 22:27:51.743640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.743607 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:51.841200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.841162 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\") pod \"dad8b200-625c-4965-9290-287e6d25be96\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " Oct 29 22:27:51.841394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.841309 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\") pod \"dad8b200-625c-4965-9290-287e6d25be96\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " Oct 29 22:27:51.841394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.841389 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\" (UniqueName: \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\") pod \"dad8b200-625c-4965-9290-287e6d25be96\" (UID: \"dad8b200-625c-4965-9290-287e6d25be96\") " Oct 29 22:27:51.842088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.842041 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca" pod "dad8b200-625c-4965-9290-287e6d25be96" (UID: "dad8b200-625c-4965-9290-287e6d25be96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:51.842286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.842258 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad8b200-625c-4965-9290-287e6d25be96-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:51.844624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.844596 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh" pod "dad8b200-625c-4965-9290-287e6d25be96" (UID: "dad8b200-625c-4965-9290-287e6d25be96"). InnerVolumeSpecName "kube-api-access-bltzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:51.845827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.845800 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca" pod "dad8b200-625c-4965-9290-287e6d25be96" (UID: "dad8b200-625c-4965-9290-287e6d25be96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:27:51.944667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.944634 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad8b200-625c-4965-9290-287e6d25be96-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:51.944807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:51.944668 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltzh\" (UniqueName: \"kubernetes.io/projected/dad8b200-625c-4965-9290-287e6d25be96-kube-api-access-bltzh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:52.215095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.213708 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" podUID="117a96c6-df8d-4db7-bfb3-23177f99e509" containerName="payload-processing" containerID="cri-o://8aa390fd0cd5755e97a610eedb193bedbcec5d4bf8944192f633e2ad2696f350" gracePeriod=25 Oct 29 22:27:52.421606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.419205 4543 generic.go:356] "Generic (PLEG): container finished" podID="117a96c6-df8d-4db7-bfb3-23177f99e509" containerID="8aa390fd0cd5755e97a610eedb193bedbcec5d4bf8944192f633e2ad2696f350" exitCode=0 Oct 29 22:27:52.421606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.419449 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" event={"ID":"117a96c6-df8d-4db7-bfb3-23177f99e509","Type":"ContainerDied","Data":"8aa390fd0cd5755e97a610eedb193bedbcec5d4bf8944192f633e2ad2696f350"} Oct 29 22:27:52.421888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.421855 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" event={"ID":"dad8b200-625c-4965-9290-287e6d25be96","Type":"ContainerDied","Data":"fa747fa24af4e79e34432bc8d5a2227a2d68edc91c9ad27397ab6461b140590e"} Oct 29 22:27:52.422018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.421912 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa747fa24af4e79e34432bc8d5a2227a2d68edc91c9ad27397ab6461b140590e" Oct 29 22:27:52.422018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.421978 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl" Oct 29 22:27:52.502760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.502736 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:52.555243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.555214 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\") pod \"117a96c6-df8d-4db7-bfb3-23177f99e509\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " Oct 29 22:27:52.556273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.555522 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\") pod \"117a96c6-df8d-4db7-bfb3-23177f99e509\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " Oct 29 22:27:52.556273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.555576 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\") pod \"117a96c6-df8d-4db7-bfb3-23177f99e509\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " Oct 29 22:27:52.556273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.555711 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\") pod \"117a96c6-df8d-4db7-bfb3-23177f99e509\" (UID: \"117a96c6-df8d-4db7-bfb3-23177f99e509\") " Oct 29 22:27:52.556273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.556191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume" pod "117a96c6-df8d-4db7-bfb3-23177f99e509" (UID: "117a96c6-df8d-4db7-bfb3-23177f99e509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:52.556693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.556669 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:52.556807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.556716 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume" pod "117a96c6-df8d-4db7-bfb3-23177f99e509" (UID: "117a96c6-df8d-4db7-bfb3-23177f99e509"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:27:52.558997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.558972 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8" pod "117a96c6-df8d-4db7-bfb3-23177f99e509" (UID: "117a96c6-df8d-4db7-bfb3-23177f99e509"). InnerVolumeSpecName "kube-api-access-l4zs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:52.559359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.559334 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials" pod "117a96c6-df8d-4db7-bfb3-23177f99e509" (UID: "117a96c6-df8d-4db7-bfb3-23177f99e509"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:52.665709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.665455 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/117a96c6-df8d-4db7-bfb3-23177f99e509-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:52.665709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.665491 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zs8\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-kube-api-access-l4zs8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:52.665709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.665505 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/117a96c6-df8d-4db7-bfb3-23177f99e509-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:27:52.708011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.707979 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:27:52.709124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709102 4543 cpu_manager.go:414] "containerMap: removing container" podUID="117a96c6-df8d-4db7-bfb3-23177f99e509" containerName="payload-processing" Oct 29 22:27:52.709124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709124 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117a96c6-df8d-4db7-bfb3-23177f99e509" containerName="payload-processing" Oct 29 22:27:52.709313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709224 4543 cpu_manager.go:414] "containerMap: removing container" podUID="dad8b200-625c-4965-9290-287e6d25be96" containerName="revoke-keys" Oct 29 22:27:52.709313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709233 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad8b200-625c-4965-9290-287e6d25be96" containerName="revoke-keys" Oct 29 22:27:52.709496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709481 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad8b200-625c-4965-9290-287e6d25be96" containerName="revoke-keys" Oct 29 22:27:52.709565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.709509 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="117a96c6-df8d-4db7-bfb3-23177f99e509" containerName="payload-processing" Oct 29 22:27:52.713836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.713816 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.733467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.733441 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:27:52.734647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.734620 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c1015\"/\"agc-cred-b1b91a0d66e7\"" Oct 29 22:27:52.766821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.766758 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.767138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.766819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.767138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.767052 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f696p\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.767138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.767102 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.868469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.868439 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.868574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.868485 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.868646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.868619 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f696p\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.868686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.868644 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.869438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.869377 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.869438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.869414 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.871239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.871218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:52.892991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:52.892966 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f696p\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\") pod \"payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:53.030373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.030316 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:53.357491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.357455 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:27:53.432765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.432743 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" Oct 29 22:27:53.432899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.432770 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5" event={"ID":"117a96c6-df8d-4db7-bfb3-23177f99e509","Type":"ContainerDied","Data":"2b5c8269f15a151aeb453058c235281505ebc3564918c6de930656c77c90776e"} Oct 29 22:27:53.432899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.432851 4543 scope.go:122] "RemoveContainer" containerID="8aa390fd0cd5755e97a610eedb193bedbcec5d4bf8944192f633e2ad2696f350" Oct 29 22:27:53.434549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.434522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" event={"ID":"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419","Type":"ContainerStarted","Data":"717d237782b73256d512cf91352648d82154803ac50c5ed54fbc1552d9aa17d9"} Oct 29 22:27:53.493413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.493389 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:53.498573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:53.498554 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-68797pzf5"] Oct 29 22:27:54.565631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:54.565594 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117a96c6-df8d-4db7-bfb3-23177f99e509" path="/var/lib/kubelet/pods/117a96c6-df8d-4db7-bfb3-23177f99e509/volumes" Oct 29 22:27:54.566244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:54.566200 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" event={"ID":"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419","Type":"ContainerStarted","Data":"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6"} Oct 29 22:27:54.566420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:54.566266 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:27:54.615664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:54.615436 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" podStartSLOduration=2.61540795 podStartE2EDuration="2.61540795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:54.613435611 +0000 UTC m=+2736.355366286" watchObservedRunningTime="2026-10-29 22:27:54.61540795 +0000 UTC m=+2736.357338624" Oct 29 22:27:54.668317 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:54.668283 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4c1015/agc-cred-b1b91a0d66e7: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:54.668317 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:54.668316 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:54.668590 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:54.668395 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials podName:3c0681a1-f3f9-4403-b3e4-06f8a3a8b419 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:55.168377302 +0000 UTC m=+2736.910307972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419") : secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:54.768312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:27:54.768283 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:27:55.188098 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:55.184816 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4c1015/agc-cred-b1b91a0d66e7: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:55.188098 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:55.184864 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:55.188098 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:55.184921 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials podName:3c0681a1-f3f9-4403-b3e4-06f8a3a8b419 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:56.184903115 +0000 UTC m=+2737.926833769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419") : secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:56.229606 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:56.229570 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4c1015/agc-cred-b1b91a0d66e7: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:56.230380 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:56.230359 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:56.230627 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:56.230612 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials podName:3c0681a1-f3f9-4403-b3e4-06f8a3a8b419 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:58.230589916 +0000 UTC m=+2739.972520584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419") : secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:57.962154 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:57.962116 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:27:58.268636 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:58.268562 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4c1015/agc-cred-b1b91a0d66e7: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:58.268636 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:58.268591 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85: secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:27:58.268814 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:27:58.268643 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials podName:3c0681a1-f3f9-4403-b3e4-06f8a3a8b419 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:02.268628575 +0000 UTC m=+2744.010559245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419") : secret "agc-cred-b1b91a0d66e7" not found Oct 29 22:28:00.234620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:00.234545 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:28:01.579990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:01.579954 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" podUID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" containerName="payload-processing" containerID="cri-o://6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6" gracePeriod=25 Oct 29 22:28:01.878443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:01.878418 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:28:02.060231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.060175 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\") pod \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " Oct 29 22:28:02.060902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.060878 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\") pod \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " Oct 29 22:28:02.062880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.061370 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume" pod "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:02.063260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.063229 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\") pod \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " Oct 29 22:28:02.086308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.074504 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\") pod \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\" (UID: \"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419\") " Oct 29 22:28:02.086308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.075445 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:02.086823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.086795 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume" pod "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:02.088334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.088232 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials" pod "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:02.090982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.090955 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p" pod "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" (UID: "3c0681a1-f3f9-4403-b3e4-06f8a3a8b419"). InnerVolumeSpecName "kube-api-access-f696p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:02.176855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.176816 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:02.176855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.176854 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:02.177060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.176865 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f696p\" (UniqueName: \"kubernetes.io/projected/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419-kube-api-access-f696p\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:02.683854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.683770 4543 generic.go:356] "Generic (PLEG): container finished" podID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" containerID="6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6" exitCode=0 Oct 29 22:28:02.683854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.683839 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" Oct 29 22:28:02.684498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.683856 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" event={"ID":"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419","Type":"ContainerDied","Data":"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6"} Oct 29 22:28:02.684498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.683910 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85" event={"ID":"3c0681a1-f3f9-4403-b3e4-06f8a3a8b419","Type":"ContainerDied","Data":"717d237782b73256d512cf91352648d82154803ac50c5ed54fbc1552d9aa17d9"} Oct 29 22:28:02.684498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.683940 4543 scope.go:122] "RemoveContainer" containerID="6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6" Oct 29 22:28:02.698180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.698158 4543 scope.go:122] "RemoveContainer" containerID="6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6" Oct 29 22:28:02.698463 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:02.698442 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6\": container with ID starting with 6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6 not found: ID does not exist" containerID="6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6" Oct 29 22:28:02.698578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.698478 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6"} err="failed to get container status \"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6\": rpc error: code = NotFound desc = could not find container \"6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6\": container with ID starting with 6054d4f886928225803dd2d8e655b15f26c45cedd520efa29130a8ae7dd039e6 not found: ID does not exist" Oct 29 22:28:02.740128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.740100 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:28:02.751551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:02.751526 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-5b6dbll85"] Oct 29 22:28:04.390118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:04.390065 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" path="/var/lib/kubelet/pods/3c0681a1-f3f9-4403-b3e4-06f8a3a8b419/volumes" Oct 29 22:28:05.237101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.236211 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" podUID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" containerName="payload-processing" containerID="cri-o://48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924" gracePeriod=25 Oct 29 22:28:05.473360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.473326 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm"] Oct 29 22:28:05.485097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.478567 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" containerName="payload-processing" Oct 29 22:28:05.485097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.478600 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" containerName="payload-processing" Oct 29 22:28:05.485097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.479385 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0681a1-f3f9-4403-b3e4-06f8a3a8b419" containerName="payload-processing" Oct 29 22:28:05.500001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.499932 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm"] Oct 29 22:28:05.500201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.500062 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.517161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.517126 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:28:05.517416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.517392 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:28:05.523410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.523385 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dsg\" (UniqueName: \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.523530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.523446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.523684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.523663 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.528591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.528570 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:28:05.528698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.528685 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:28:05.567957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.567898 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:28:05.624600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.624571 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\") pod \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " Oct 29 22:28:05.624740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.624652 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\") pod \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " Oct 29 22:28:05.624874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.624850 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\") pod \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " Oct 29 22:28:05.625225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.625166 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume" pod "dc77a3b0-8321-4632-aa0b-b2d82b66953b" (UID: "dc77a3b0-8321-4632-aa0b-b2d82b66953b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:05.625335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.625224 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume" pod "dc77a3b0-8321-4632-aa0b-b2d82b66953b" (UID: "dc77a3b0-8321-4632-aa0b-b2d82b66953b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:05.625442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.625172 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\") pod \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\" (UID: \"dc77a3b0-8321-4632-aa0b-b2d82b66953b\") " Oct 29 22:28:05.625809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.625785 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dsg\" (UniqueName: \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.625990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.625970 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.626222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.626201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.626539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.626518 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:05.626676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.626660 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc77a3b0-8321-4632-aa0b-b2d82b66953b-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:05.627109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.626898 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.628175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.628144 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn" pod "dc77a3b0-8321-4632-aa0b-b2d82b66953b" (UID: "dc77a3b0-8321-4632-aa0b-b2d82b66953b"). InnerVolumeSpecName "kube-api-access-vb6cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:05.628338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.628315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials" pod "dc77a3b0-8321-4632-aa0b-b2d82b66953b" (UID: "dc77a3b0-8321-4632-aa0b-b2d82b66953b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:05.629262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.629236 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.650303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.650282 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dsg\" (UniqueName: \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\") pod \"maas-api-revoke-sub-dab65700-phxvm\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:05.716850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.716808 4543 generic.go:356] "Generic (PLEG): container finished" podID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" containerID="48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924" exitCode=0 Oct 29 22:28:05.717175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.717150 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" event={"ID":"dc77a3b0-8321-4632-aa0b-b2d82b66953b","Type":"ContainerDied","Data":"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924"} Oct 29 22:28:05.717273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.717206 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" event={"ID":"dc77a3b0-8321-4632-aa0b-b2d82b66953b","Type":"ContainerDied","Data":"fd0910e0d8e5d1004bcca817a7108127f0a025f64678729cd5314886fb73c2bd"} Oct 29 22:28:05.717273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.717237 4543 scope.go:122] "RemoveContainer" containerID="48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924" Oct 29 22:28:05.717401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.717387 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25" Oct 29 22:28:05.728323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.728303 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:05.728323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.728324 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6cn\" (UniqueName: \"kubernetes.io/projected/dc77a3b0-8321-4632-aa0b-b2d82b66953b-kube-api-access-vb6cn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:05.734577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.734558 4543 scope.go:122] "RemoveContainer" containerID="48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924" Oct 29 22:28:05.734875 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:05.734851 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924\": container with ID starting with 48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924 not found: ID does not exist" containerID="48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924" Oct 29 22:28:05.734960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.734881 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924"} err="failed to get container status \"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924\": rpc error: code = NotFound desc = could not find container \"48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924\": container with ID starting with 48293e5530dd51425c4bc0e6c3e6eda1f6544d3544e7ace2776d3fc3855bf924 not found: ID does not exist" Oct 29 22:28:05.777403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.777354 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:28:05.782448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.782429 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b7c2d5/payload-processing-external-model-e2e-extmod-a-b7c2d5-5c8dlvf25"] Oct 29 22:28:05.824638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:05.824618 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:06.139472 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:06.139441 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ccc2c8_a104_49e8_8793_19e087e8e368.slice/crio-0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f WatchSource:0}: Error finding container 0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f: Status 404 returned error can't find the container with id 0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f Oct 29 22:28:06.139812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:06.139790 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm"] Oct 29 22:28:06.388893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:06.388837 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" path="/var/lib/kubelet/pods/dc77a3b0-8321-4632-aa0b-b2d82b66953b/volumes" Oct 29 22:28:06.729454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:06.729423 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" event={"ID":"94ccc2c8-a104-49e8-8793-19e087e8e368","Type":"ContainerStarted","Data":"8e434f29347737b2cec22886faaf2364717647737fa3a72b148b140b06beeda6"} Oct 29 22:28:06.729887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:06.729481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" event={"ID":"94ccc2c8-a104-49e8-8793-19e087e8e368","Type":"ContainerStarted","Data":"0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f"} Oct 29 22:28:06.773048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:06.773006 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" podStartSLOduration=1.772993605 podStartE2EDuration="1.772993605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:06.770531027 +0000 UTC m=+2748.512461701" watchObservedRunningTime="2026-10-29 22:28:06.772993605 +0000 UTC m=+2748.514924314" Oct 29 22:28:07.739844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:07.739810 4543 generic.go:356] "Generic (PLEG): container finished" podID="94ccc2c8-a104-49e8-8793-19e087e8e368" containerID="8e434f29347737b2cec22886faaf2364717647737fa3a72b148b140b06beeda6" exitCode=0 Oct 29 22:28:07.740243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:07.739892 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" event={"ID":"94ccc2c8-a104-49e8-8793-19e087e8e368","Type":"ContainerDied","Data":"8e434f29347737b2cec22886faaf2364717647737fa3a72b148b140b06beeda6"} Oct 29 22:28:08.943178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.943151 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:08.969753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.969729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\" (UniqueName: \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\") pod \"94ccc2c8-a104-49e8-8793-19e087e8e368\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " Oct 29 22:28:08.969922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.969908 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\") pod \"94ccc2c8-a104-49e8-8793-19e087e8e368\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " Oct 29 22:28:08.970006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.969992 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\") pod \"94ccc2c8-a104-49e8-8793-19e087e8e368\" (UID: \"94ccc2c8-a104-49e8-8793-19e087e8e368\") " Oct 29 22:28:08.970318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.970285 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca" pod "94ccc2c8-a104-49e8-8793-19e087e8e368" (UID: "94ccc2c8-a104-49e8-8793-19e087e8e368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:08.970673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.970654 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:08.972027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.972002 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg" pod "94ccc2c8-a104-49e8-8793-19e087e8e368" (UID: "94ccc2c8-a104-49e8-8793-19e087e8e368"). InnerVolumeSpecName "kube-api-access-97dsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:08.972175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:08.972069 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca" pod "94ccc2c8-a104-49e8-8793-19e087e8e368" (UID: "94ccc2c8-a104-49e8-8793-19e087e8e368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:28:09.071764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:09.071694 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94ccc2c8-a104-49e8-8793-19e087e8e368-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:09.071764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:09.071722 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dsg\" (UniqueName: \"kubernetes.io/projected/94ccc2c8-a104-49e8-8793-19e087e8e368-kube-api-access-97dsg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:09.760966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:09.760935 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" Oct 29 22:28:09.761169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:09.760970 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm" event={"ID":"94ccc2c8-a104-49e8-8793-19e087e8e368","Type":"ContainerDied","Data":"0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f"} Oct 29 22:28:09.761169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:09.761023 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb9341efaef09e8a5e684c56f84e4c13db1ff56d4f8c94ba3e8a0e9f657cb9f" Oct 29 22:28:10.060635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:10.060534 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr"] Oct 29 22:28:10.066455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:10.066428 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d677cb43-gbxlr"] Oct 29 22:28:10.389209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:10.389133 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3856621b-4b8a-4e64-928a-3394b6a2c54c" path="/var/lib/kubelet/pods/3856621b-4b8a-4e64-928a-3394b6a2c54c/volumes" Oct 29 22:28:18.368307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.368278 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:18.369734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.369711 4543 cpu_manager.go:414] "containerMap: removing container" podUID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" containerName="payload-processing" Oct 29 22:28:18.369792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.369738 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" containerName="payload-processing" Oct 29 22:28:18.369792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.369753 4543 cpu_manager.go:414] "containerMap: removing container" podUID="94ccc2c8-a104-49e8-8793-19e087e8e368" containerName="revoke-keys" Oct 29 22:28:18.369792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.369761 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ccc2c8-a104-49e8-8793-19e087e8e368" containerName="revoke-keys" Oct 29 22:28:18.370012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.369997 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc77a3b0-8321-4632-aa0b-b2d82b66953b" containerName="payload-processing" Oct 29 22:28:18.370064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.370019 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ccc2c8-a104-49e8-8793-19e087e8e368" containerName="revoke-keys" Oct 29 22:28:18.373664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.373647 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.401499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401476 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d8760\"" Oct 29 22:28:18.401631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401556 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"routing-overlay\"" Oct 29 22:28:18.401631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401567 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"agc-cred-c2e97682144e\"" Oct 29 22:28:18.401631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401586 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"kube-root-ca.crt\"" Oct 29 22:28:18.401631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401589 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"openshift-service-ca.crt\"" Oct 29 22:28:18.401787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.401636 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-e2e--e23b4f7e-dockercfg-ftsnc\"" Oct 29 22:28:18.407998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.407970 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:18.455459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.455434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.455678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.455661 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsrx\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.455866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.455850 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.456010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.455989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.564118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.557421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.564118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.557676 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsrx\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.564118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.557854 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.564118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.557941 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.585099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.577386 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d8760\"" Oct 29 22:28:18.585099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.577488 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"routing-overlay\"" Oct 29 22:28:18.585099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.577553 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"agc-cred-c2e97682144e\"" Oct 29 22:28:18.607096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.595522 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.607096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.596044 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.607096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.600099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.617144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.608340 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"kube-root-ca.crt\"" Oct 29 22:28:18.644142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.643296 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"openshift-service-ca.crt\"" Oct 29 22:28:18.675461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.657887 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsrx\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\") pod \"payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.721723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.719431 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d8760\"/\"payload-processing-external-model-e2e--e23b4f7e-dockercfg-ftsnc\"" Oct 29 22:28:18.726090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.724860 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:18.869095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:18.868881 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:28:19.258526 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:19.258500 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa40abed_2b1e_467f_bd44_98412b641c93.slice/crio-e39b47d2e86b34efb4c827ebf1a7ca047128c008a970a802ae5b573d24d6fde9 WatchSource:0}: Error finding container e39b47d2e86b34efb4c827ebf1a7ca047128c008a970a802ae5b573d24d6fde9: Status 404 returned error can't find the container with id e39b47d2e86b34efb4c827ebf1a7ca047128c008a970a802ae5b573d24d6fde9 Oct 29 22:28:19.259565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:19.259513 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:19.990597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:19.990556 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" event={"ID":"aa40abed-2b1e-467f-bd44-98412b641c93","Type":"ContainerStarted","Data":"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c"} Oct 29 22:28:19.991005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:19.990625 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" event={"ID":"aa40abed-2b1e-467f-bd44-98412b641c93","Type":"ContainerStarted","Data":"e39b47d2e86b34efb4c827ebf1a7ca047128c008a970a802ae5b573d24d6fde9"} Oct 29 22:28:19.991005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:19.990657 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:20.044194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:20.044139 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" podStartSLOduration=2.044121692 podStartE2EDuration="2.044121692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:20.037541833 +0000 UTC m=+2761.779472505" watchObservedRunningTime="2026-10-29 22:28:20.044121692 +0000 UTC m=+2761.786052367" Oct 29 22:28:20.112051 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.112023 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d8760/agc-cred-c2e97682144e: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.112051 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.112049 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.112351 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.112146 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:20.61212804 +0000 UTC m=+2762.354058705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.637584 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.637551 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d8760/agc-cred-c2e97682144e: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.637584 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.637575 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.637805 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:20.637632 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:21.637618458 +0000 UTC m=+2763.379549109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : secret "agc-cred-c2e97682144e" not found Oct 29 22:28:20.843771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:20.843733 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:21.001613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:21.001580 4543 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" secret="" err="secret \"payload-processing-external-model-e2e--e23b4f7e-dockercfg-ftsnc\" not found" Oct 29 22:28:21.147878 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.147848 4543 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-plugins-e2e-extmod-b-7d8760: configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:21.148066 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.147919 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:21.647905842 +0000 UTC m=+2763.389836513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:21.659389 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.659360 4543 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-plugins-e2e-extmod-b-7d8760: configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:21.659566 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.659422 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:22.659409059 +0000 UTC m=+2764.401339713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:21.659566 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.659470 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d8760/agc-cred-c2e97682144e: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:21.659566 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.659494 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:21.659566 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:21.659548 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:23.659536391 +0000 UTC m=+2765.401467042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : secret "agc-cred-c2e97682144e" not found Oct 29 22:28:22.673764 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:22.673730 4543 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-plugins-e2e-extmod-b-7d8760: configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:22.674273 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:22.673823 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:24.673801633 +0000 UTC m=+2766.415732283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:23.359234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.359204 4543 scope.go:122] "RemoveContainer" containerID="b222f11c86cdfda4019ecd901e3aedf5a0195725ce5d57018dc33805b5ec28e3" Oct 29 22:28:23.371167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.371147 4543 scope.go:122] "RemoveContainer" containerID="c7515cb966fd77f0085576f14cd8f8d8ccdfd27cf6b488cf187f4a7024db2e30" Oct 29 22:28:23.382524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.382503 4543 scope.go:122] "RemoveContainer" containerID="f8816485831c8dce0cecb6f5318d11b99b89e420c5e73da05b52f238bb50b69b" Oct 29 22:28:23.392618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.392600 4543 scope.go:122] "RemoveContainer" containerID="39c3fd1584fd4dd988b68a3f4863637009796f60168ed6110c28eb3a7add50a3" Oct 29 22:28:23.403766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.403744 4543 scope.go:122] "RemoveContainer" containerID="cf6dbdc2ec0c73c241087899353d8a989c99430dafc545892764575c670ab94e" Oct 29 22:28:23.686484 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:23.686419 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d8760/agc-cred-c2e97682144e: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:23.686484 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:23.686438 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt: secret "agc-cred-c2e97682144e" not found Oct 29 22:28:23.686872 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:23.686490 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:27.686477484 +0000 UTC m=+2769.428408155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : secret "agc-cred-c2e97682144e" not found Oct 29 22:28:23.869575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:23.869536 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" podUID="9ca6b60e-058a-4da4-a363-356faa120905" containerName="payload-processing" containerID="cri-o://d1e42af1e310c213d851d78f1b4ac79156c9625e7af2c3d88047deccb8f95c92" gracePeriod=25 Oct 29 22:28:24.080434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.080384 4543 generic.go:356] "Generic (PLEG): container finished" podID="9ca6b60e-058a-4da4-a363-356faa120905" containerID="d1e42af1e310c213d851d78f1b4ac79156c9625e7af2c3d88047deccb8f95c92" exitCode=0 Oct 29 22:28:24.080758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.080740 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" event={"ID":"9ca6b60e-058a-4da4-a363-356faa120905","Type":"ContainerDied","Data":"d1e42af1e310c213d851d78f1b4ac79156c9625e7af2c3d88047deccb8f95c92"} Oct 29 22:28:24.229423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.229397 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:28:24.296675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.296602 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\") pod \"9ca6b60e-058a-4da4-a363-356faa120905\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " Oct 29 22:28:24.296908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.296886 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\") pod \"9ca6b60e-058a-4da4-a363-356faa120905\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " Oct 29 22:28:24.296978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.296951 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\") pod \"9ca6b60e-058a-4da4-a363-356faa120905\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " Oct 29 22:28:24.297057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.296979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\") pod \"9ca6b60e-058a-4da4-a363-356faa120905\" (UID: \"9ca6b60e-058a-4da4-a363-356faa120905\") " Oct 29 22:28:24.297057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.297006 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume" pod "9ca6b60e-058a-4da4-a363-356faa120905" (UID: "9ca6b60e-058a-4da4-a363-356faa120905"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:24.297492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.297460 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume" pod "9ca6b60e-058a-4da4-a363-356faa120905" (UID: "9ca6b60e-058a-4da4-a363-356faa120905"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:24.298025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.297994 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:24.298163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.298031 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca6b60e-058a-4da4-a363-356faa120905-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:24.299968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.299942 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials" pod "9ca6b60e-058a-4da4-a363-356faa120905" (UID: "9ca6b60e-058a-4da4-a363-356faa120905"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:24.300292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.300272 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p" pod "9ca6b60e-058a-4da4-a363-356faa120905" (UID: "9ca6b60e-058a-4da4-a363-356faa120905"). InnerVolumeSpecName "kube-api-access-tkh4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:24.399246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.399225 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:24.399339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:24.399248 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkh4p\" (UniqueName: \"kubernetes.io/projected/9ca6b60e-058a-4da4-a363-356faa120905-kube-api-access-tkh4p\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:24.703424 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:24.703361 4543 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-plugins-e2e-extmod-b-7d8760: configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:24.703751 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:24.703432 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume podName:aa40abed-2b1e-467f-bd44-98412b641c93 nodeName:}" failed. No retries permitted until 2026-10-29 22:28:28.70341928 +0000 UTC m=+2770.445349945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-7d8760" not found Oct 29 22:28:25.091959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:25.091925 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" event={"ID":"9ca6b60e-058a-4da4-a363-356faa120905","Type":"ContainerDied","Data":"8531b531fbca79e5b5b9d7c72c3b4dbb5bb6a9779d912539ebe8cf2ba502c953"} Oct 29 22:28:25.092138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:25.091960 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs" Oct 29 22:28:25.092138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:25.092002 4543 scope.go:122] "RemoveContainer" containerID="d1e42af1e310c213d851d78f1b4ac79156c9625e7af2c3d88047deccb8f95c92" Oct 29 22:28:25.151880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:25.151848 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:28:25.164585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:25.164563 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c1015/payload-processing-external-model-e2e-extmod-b-4c1015-dcffq8kvs"] Oct 29 22:28:26.389207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:26.389175 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca6b60e-058a-4da4-a363-356faa120905" path="/var/lib/kubelet/pods/9ca6b60e-058a-4da4-a363-356faa120905/volumes" Oct 29 22:28:27.012019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.011983 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" podUID="aa40abed-2b1e-467f-bd44-98412b641c93" containerName="payload-processing" containerID="cri-o://57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c" gracePeriod=25 Oct 29 22:28:27.370758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.370730 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:27.443397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.443362 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\") pod \"aa40abed-2b1e-467f-bd44-98412b641c93\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " Oct 29 22:28:27.443817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.443513 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\") pod \"aa40abed-2b1e-467f-bd44-98412b641c93\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " Oct 29 22:28:27.443817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.443720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\") pod \"aa40abed-2b1e-467f-bd44-98412b641c93\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " Oct 29 22:28:27.443963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.443812 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume" pod "aa40abed-2b1e-467f-bd44-98412b641c93" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:27.443963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.443893 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\") pod \"aa40abed-2b1e-467f-bd44-98412b641c93\" (UID: \"aa40abed-2b1e-467f-bd44-98412b641c93\") " Oct 29 22:28:27.444957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.444808 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:27.445089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.444962 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume" pod "aa40abed-2b1e-467f-bd44-98412b641c93" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:28:27.447295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.447269 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials" pod "aa40abed-2b1e-467f-bd44-98412b641c93" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:27.447404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.447387 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx" pod "aa40abed-2b1e-467f-bd44-98412b641c93" (UID: "aa40abed-2b1e-467f-bd44-98412b641c93"). InnerVolumeSpecName "kube-api-access-xjsrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:28:27.546395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.546360 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa40abed-2b1e-467f-bd44-98412b641c93-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:27.546395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.546389 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsrx\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-kube-api-access-xjsrx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:27.546395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:27.546401 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa40abed-2b1e-467f-bd44-98412b641c93-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:28:27.962214 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:27.962178 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:28:28.123982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.123951 4543 generic.go:356] "Generic (PLEG): container finished" podID="aa40abed-2b1e-467f-bd44-98412b641c93" containerID="57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c" exitCode=0 Oct 29 22:28:28.124158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.123993 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" Oct 29 22:28:28.124158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.123994 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" event={"ID":"aa40abed-2b1e-467f-bd44-98412b641c93","Type":"ContainerDied","Data":"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c"} Oct 29 22:28:28.124158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.124139 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt" event={"ID":"aa40abed-2b1e-467f-bd44-98412b641c93","Type":"ContainerDied","Data":"e39b47d2e86b34efb4c827ebf1a7ca047128c008a970a802ae5b573d24d6fde9"} Oct 29 22:28:28.124287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.124173 4543 scope.go:122] "RemoveContainer" containerID="57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c" Oct 29 22:28:28.136003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.135982 4543 scope.go:122] "RemoveContainer" containerID="57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c" Oct 29 22:28:28.136334 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:28.136307 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c\": container with ID starting with 57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c not found: ID does not exist" containerID="57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c" Oct 29 22:28:28.136392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.136346 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c"} err="failed to get container status \"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c\": rpc error: code = NotFound desc = could not find container \"57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c\": container with ID starting with 57b325b648d0581f6a89d5dd5e73fc29d0f02384659db48032113a44d0c3e50c not found: ID does not exist" Oct 29 22:28:28.163834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.163803 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:28:28.165380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165351 4543 cpu_manager.go:414] "containerMap: removing container" podUID="aa40abed-2b1e-467f-bd44-98412b641c93" containerName="payload-processing" Oct 29 22:28:28.165380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165377 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa40abed-2b1e-467f-bd44-98412b641c93" containerName="payload-processing" Oct 29 22:28:28.165555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165429 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9ca6b60e-058a-4da4-a363-356faa120905" containerName="payload-processing" Oct 29 22:28:28.165555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165437 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca6b60e-058a-4da4-a363-356faa120905" containerName="payload-processing" Oct 29 22:28:28.165667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165655 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa40abed-2b1e-467f-bd44-98412b641c93" containerName="payload-processing" Oct 29 22:28:28.165722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.165673 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca6b60e-058a-4da4-a363-356faa120905" containerName="payload-processing" Oct 29 22:28:28.208124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.208095 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.209612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.209586 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:28.216149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.216065 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:28:28.222561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.222533 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d8760/payload-processing-external-model-e2e-extmod-b-7d8760-77796dcnt"] Oct 29 22:28:28.226127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.226069 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c06a2c-openshift-default-dockercfg-fj64d\"" Oct 29 22:28:28.259214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259183 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259231 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259271 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259402 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259445 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259571 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259687 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.259938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.259913 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwdd\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376369 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376483 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376712 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwdd\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376907 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376941 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.377104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.376988 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.423101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.422563 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.423586 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.423816 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.423845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.424175 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.435361 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.503161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.493423 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.511109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.508108 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwdd\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.518975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.518941 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa40abed-2b1e-467f-bd44-98412b641c93" path="/var/lib/kubelet/pods/aa40abed-2b1e-467f-bd44-98412b641c93/volumes" Oct 29 22:28:28.521387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.521364 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\") pod \"e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.536807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.536738 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:28.953909 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:28.953877 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac07f82_9cbd_446e_80f6_c8a1b4202005.slice/crio-6f006a90c0b7464a157010f99a3652b0d12df7b697e96f5a10cd3a60bd2b94eb WatchSource:0}: Error finding container 6f006a90c0b7464a157010f99a3652b0d12df7b697e96f5a10cd3a60bd2b94eb: Status 404 returned error can't find the container with id 6f006a90c0b7464a157010f99a3652b0d12df7b697e96f5a10cd3a60bd2b94eb Oct 29 22:28:28.954869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.954842 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:28:28.957047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.957008 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:28.957187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.957107 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:28.957187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:28.957157 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:29.137496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:29.137461 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerStarted","Data":"177453861d86c20095c57375c07acd5cc3e6d96dbb79749d98188d5b6b241009"} Oct 29 22:28:29.137702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:29.137515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerStarted","Data":"6f006a90c0b7464a157010f99a3652b0d12df7b697e96f5a10cd3a60bd2b94eb"} Oct 29 22:28:29.197097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:29.193665 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podStartSLOduration=1.193646938 podStartE2EDuration="1.193646938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:29.188021876 +0000 UTC m=+2770.929952551" watchObservedRunningTime="2026-10-29 22:28:29.193646938 +0000 UTC m=+2770.935577612" Oct 29 22:28:29.537818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:29.537791 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:30.544205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:30.544175 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:31.155616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:31.155589 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:31.156517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:31.156496 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:28:33.698813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:33.698778 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:28:33.953149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:33.952687 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:28:33.953149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:33.952850 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:33.970421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:33.970393 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c06a2c\"" Oct 29 22:28:33.983635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:33.983602 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c06a2c-dockercfg-59v44\"" Oct 29 22:28:34.040736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.040710 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6tg\" (UniqueName: \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.041041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.041019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.142412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.142389 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.142528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.142497 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6tg\" (UniqueName: \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.143148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.143130 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.168496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.168471 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6tg\" (UniqueName: \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\") pod \"payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.278296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.278273 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:34.595297 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:34.595269 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51c89c9_aebd_40b1_927f_9dbe7b85db51.slice/crio-40799d57093bc639f39978aa1b58b0c6cc6cba4268a8549090ede6429d61016e WatchSource:0}: Error finding container 40799d57093bc639f39978aa1b58b0c6cc6cba4268a8549090ede6429d61016e: Status 404 returned error can't find the container with id 40799d57093bc639f39978aa1b58b0c6cc6cba4268a8549090ede6429d61016e Oct 29 22:28:34.596872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:34.596783 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:28:35.196596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:35.196557 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" event={"ID":"c51c89c9-aebd-40b1-927f-9dbe7b85db51","Type":"ContainerStarted","Data":"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462"} Oct 29 22:28:35.197189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:35.196618 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" event={"ID":"c51c89c9-aebd-40b1-927f-9dbe7b85db51","Type":"ContainerStarted","Data":"40799d57093bc639f39978aa1b58b0c6cc6cba4268a8549090ede6429d61016e"} Oct 29 22:28:35.197189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:35.196645 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:35.247769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:35.247705 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" podStartSLOduration=2.247685735 podStartE2EDuration="2.247685735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:35.241113794 +0000 UTC m=+2776.983044467" watchObservedRunningTime="2026-10-29 22:28:35.247685735 +0000 UTC m=+2776.989616409" Oct 29 22:28:39.960727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:39.960697 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:28:39.967512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:39.967461 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:39.998634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:39.998607 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:28:40.000546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.000524 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-201a3d-openshift-default-dockercfg-pjh7f\"" Oct 29 22:28:40.061289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061255 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061413 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w798f\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061587 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061669 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061720 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061764 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.061895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.061885 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175474 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175610 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175647 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175757 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.175899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175818 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w798f\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.176014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175895 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.176014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.175942 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.177132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.177019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.177316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.177284 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.177763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.177729 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.198792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.191744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.198792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.192280 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.199011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.198875 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.215883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.209230 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.249099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.237141 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w798f\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.249099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.248661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\") pod \"e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.323115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.316398 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:40.799188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.799156 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:28:40.799392 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:40.799366 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b613ca_f1c4_4764_b23d_f98afd462fa4.slice/crio-934512e213055df533863262a040d902b6df69fa7014a5bef886c22a03a8f6b3 WatchSource:0}: Error finding container 934512e213055df533863262a040d902b6df69fa7014a5bef886c22a03a8f6b3: Status 404 returned error can't find the container with id 934512e213055df533863262a040d902b6df69fa7014a5bef886c22a03a8f6b3 Oct 29 22:28:40.802789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.802755 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:40.802873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.802826 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:40.802873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:40.802858 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:28:41.337099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.332147 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerStarted","Data":"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5"} Oct 29 22:28:41.337099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.332306 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerStarted","Data":"934512e213055df533863262a040d902b6df69fa7014a5bef886c22a03a8f6b3"} Oct 29 22:28:41.413544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.413489 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podStartSLOduration=2.413476405 podStartE2EDuration="2.413476405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:41.394421754 +0000 UTC m=+2783.136352427" watchObservedRunningTime="2026-10-29 22:28:41.413476405 +0000 UTC m=+2783.155407079" Oct 29 22:28:41.669407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.669326 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:28:41.674716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.674695 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:41.692359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.692340 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-75ea0c\"" Oct 29 22:28:41.692540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.692512 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-75ea0c-dockercfg-fzcsn\"" Oct 29 22:28:41.692740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.692720 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:28:41.822100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.806644 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr77t\" (UniqueName: \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:41.822100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.806938 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:41.909101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.893310 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:28:41.921222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.916346 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:41.921222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.916596 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr77t\" (UniqueName: \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:41.921222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:41.918503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:42.037100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.036600 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.055099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.044451 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr77t\" (UniqueName: \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\") pod \"payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:42.162934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.162854 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:28:42.257273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.257236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllz4\" (UniqueName: \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.257472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.257395 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.291279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.291255 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:42.317860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.317835 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:42.325019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.324995 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:42.342503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.342476 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:42.344645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.344618 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:28:42.362557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.362528 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.363059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.363028 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllz4\" (UniqueName: \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.363668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.363628 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.392256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.392228 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllz4\" (UniqueName: \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\") pod \"payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.482672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.482640 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:42.612145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.612113 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:28:42.612739 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:42.612714 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde465347_1a0b_4c58_9a53_6c78fb2e8b19.slice/crio-6f3b40a242be20630e86d4573b9265b76ac76ade4e7f5f8ff22d9bca7257bbef WatchSource:0}: Error finding container 6f3b40a242be20630e86d4573b9265b76ac76ade4e7f5f8ff22d9bca7257bbef: Status 404 returned error can't find the container with id 6f3b40a242be20630e86d4573b9265b76ac76ade4e7f5f8ff22d9bca7257bbef Oct 29 22:28:42.809154 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:42.809126 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf5d693_1097_4cbb_a0b5_e8ecb6e3e239.slice/crio-a395af450ae9c1940beedc5a041b108dcc42b5efce2977843f09dbdcfe383091 WatchSource:0}: Error finding container a395af450ae9c1940beedc5a041b108dcc42b5efce2977843f09dbdcfe383091: Status 404 returned error can't find the container with id a395af450ae9c1940beedc5a041b108dcc42b5efce2977843f09dbdcfe383091 Oct 29 22:28:42.810395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:42.810319 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:28:43.353648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.353617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" event={"ID":"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239","Type":"ContainerStarted","Data":"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766"} Oct 29 22:28:43.354211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.353674 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" event={"ID":"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239","Type":"ContainerStarted","Data":"a395af450ae9c1940beedc5a041b108dcc42b5efce2977843f09dbdcfe383091"} Oct 29 22:28:43.354211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.353702 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:43.355605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.355580 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" event={"ID":"de465347-1a0b-4c58-9a53-6c78fb2e8b19","Type":"ContainerStarted","Data":"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459"} Oct 29 22:28:43.355702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.355624 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" event={"ID":"de465347-1a0b-4c58-9a53-6c78fb2e8b19","Type":"ContainerStarted","Data":"6f3b40a242be20630e86d4573b9265b76ac76ade4e7f5f8ff22d9bca7257bbef"} Oct 29 22:28:43.355810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.355800 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:43.402879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.402836 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" podStartSLOduration=2.402822046 podStartE2EDuration="2.402822046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:43.399297155 +0000 UTC m=+2785.141227827" watchObservedRunningTime="2026-10-29 22:28:43.402822046 +0000 UTC m=+2785.144752718" Oct 29 22:28:43.443999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:43.443956 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" podStartSLOduration=2.443940857 podStartE2EDuration="2.443940857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:43.441853294 +0000 UTC m=+2785.183783966" watchObservedRunningTime="2026-10-29 22:28:43.443940857 +0000 UTC m=+2785.185871531" Oct 29 22:28:45.982589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:45.982551 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:28:45.989084 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:45.989049 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.003769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.003743 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:28:46.010025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.010003 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-201a3d-dockercfg-zcrtz\"" Oct 29 22:28:46.010180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.010062 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-201a3d\"" Oct 29 22:28:46.026645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.026621 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcc6\" (UniqueName: \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.027028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.027006 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.148101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.147654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.148101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.147799 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcc6\" (UniqueName: \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.164103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.149226 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.246153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.245607 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:28:46.286101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.268468 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcc6\" (UniqueName: \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\") pod \"payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.318102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.310891 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:46.397168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.347799 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:28:46.397168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.348002 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.397168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.351116 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:28:46.418105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.407120 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.418105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.407181 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ttr\" (UniqueName: \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.509344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.509274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.509344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.509319 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ttr\" (UniqueName: \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.510297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.510271 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.535984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.535949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ttr\" (UniqueName: \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\") pod \"payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:46.778608 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:46.778573 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcd68a9_903b_4c02_83be_baf039321971.slice/crio-ae490805a41b9d18afddfd084123fafb4399b21997f984f2d8e06d930bf35f6f WatchSource:0}: Error finding container ae490805a41b9d18afddfd084123fafb4399b21997f984f2d8e06d930bf35f6f: Status 404 returned error can't find the container with id ae490805a41b9d18afddfd084123fafb4399b21997f984f2d8e06d930bf35f6f Oct 29 22:28:46.779184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.779131 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:28:46.796387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:46.796356 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:47.150728 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:28:47.150675 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d5410e_d4e5_4e4f_b399_4d42ab459d02.slice/crio-b2f592f3da9fa9e724243cd27eb7b9779f9ed1e4882afe81b681b7ee760f08c9 WatchSource:0}: Error finding container b2f592f3da9fa9e724243cd27eb7b9779f9ed1e4882afe81b681b7ee760f08c9: Status 404 returned error can't find the container with id b2f592f3da9fa9e724243cd27eb7b9779f9ed1e4882afe81b681b7ee760f08c9 Oct 29 22:28:47.151400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.151305 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:28:47.470400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.470362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" event={"ID":"cfcd68a9-903b-4c02-83be-baf039321971","Type":"ContainerStarted","Data":"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589"} Oct 29 22:28:47.470566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.470418 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" event={"ID":"cfcd68a9-903b-4c02-83be-baf039321971","Type":"ContainerStarted","Data":"ae490805a41b9d18afddfd084123fafb4399b21997f984f2d8e06d930bf35f6f"} Oct 29 22:28:47.470566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.470480 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:28:47.472338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.472315 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" event={"ID":"57d5410e-d4e5-4e4f-b399-4d42ab459d02","Type":"ContainerStarted","Data":"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58"} Oct 29 22:28:47.472459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.472353 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" event={"ID":"57d5410e-d4e5-4e4f-b399-4d42ab459d02","Type":"ContainerStarted","Data":"b2f592f3da9fa9e724243cd27eb7b9779f9ed1e4882afe81b681b7ee760f08c9"} Oct 29 22:28:47.472459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.472389 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:47.513896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.513846 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" podStartSLOduration=2.513830727 podStartE2EDuration="2.513830727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:47.513670249 +0000 UTC m=+2789.255600934" watchObservedRunningTime="2026-10-29 22:28:47.513830727 +0000 UTC m=+2789.255761402" Oct 29 22:28:47.559728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:47.559685 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" podStartSLOduration=1.559673213 podStartE2EDuration="1.559673213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:47.557451316 +0000 UTC m=+2789.299381991" watchObservedRunningTime="2026-10-29 22:28:47.559673213 +0000 UTC m=+2789.301603937" Oct 29 22:28:54.368145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:54.368111 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:28:54.368676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:54.368509 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:28:57.964013 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:28:57.963979 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:28:58.484791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:58.484759 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:28:58.485134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:28:58.485116 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:29:08.524639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.524609 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5"] Oct 29 22:29:08.531137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.531120 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.548006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.547983 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5"] Oct 29 22:29:08.554516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.554467 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:29:08.554738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.554712 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:29:08.554869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.554856 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:29:08.560579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.560558 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:29:08.605302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.605278 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.605494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.605475 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv28k\" (UniqueName: \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.605621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.605608 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.707317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.707295 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv28k\" (UniqueName: \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.707420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.707388 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.707497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.707486 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.708027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.708007 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.710232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.710213 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.732837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.732809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv28k\" (UniqueName: \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\") pod \"maas-api-revoke-sub-959407d4-r2sx5\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:08.847623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:08.847553 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:09.169719 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:29:09.169676 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08530db3_b66e_441b_bbfe_1a77037e2fd4.slice/crio-665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1 WatchSource:0}: Error finding container 665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1: Status 404 returned error can't find the container with id 665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1 Oct 29 22:29:09.170160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:09.170139 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5"] Oct 29 22:29:09.808491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:09.808420 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" event={"ID":"08530db3-b66e-441b-bbfe-1a77037e2fd4","Type":"ContainerStarted","Data":"80c114b30a4c16a55df39ea035c36f6857db5956c2c40f792948243143ffe04b"} Oct 29 22:29:09.808491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:09.808465 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" event={"ID":"08530db3-b66e-441b-bbfe-1a77037e2fd4","Type":"ContainerStarted","Data":"665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1"} Oct 29 22:29:09.865767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:09.865716 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" podStartSLOduration=1.865701166 podStartE2EDuration="1.865701166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:09.85747769 +0000 UTC m=+2811.599408364" watchObservedRunningTime="2026-10-29 22:29:09.865701166 +0000 UTC m=+2811.607631838" Oct 29 22:29:10.819884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:10.819846 4543 generic.go:356] "Generic (PLEG): container finished" podID="08530db3-b66e-441b-bbfe-1a77037e2fd4" containerID="80c114b30a4c16a55df39ea035c36f6857db5956c2c40f792948243143ffe04b" exitCode=0 Oct 29 22:29:10.820291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:10.819914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" event={"ID":"08530db3-b66e-441b-bbfe-1a77037e2fd4","Type":"ContainerDied","Data":"80c114b30a4c16a55df39ea035c36f6857db5956c2c40f792948243143ffe04b"} Oct 29 22:29:12.021953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.021928 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:12.181138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.181023 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\") pod \"08530db3-b66e-441b-bbfe-1a77037e2fd4\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " Oct 29 22:29:12.181138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.181134 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\") pod \"08530db3-b66e-441b-bbfe-1a77037e2fd4\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " Oct 29 22:29:12.181395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.181362 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\" (UniqueName: \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\") pod \"08530db3-b66e-441b-bbfe-1a77037e2fd4\" (UID: \"08530db3-b66e-441b-bbfe-1a77037e2fd4\") " Oct 29 22:29:12.181626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.181594 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca" pod "08530db3-b66e-441b-bbfe-1a77037e2fd4" (UID: "08530db3-b66e-441b-bbfe-1a77037e2fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:12.182482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.182456 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:12.183625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.183594 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca" pod "08530db3-b66e-441b-bbfe-1a77037e2fd4" (UID: "08530db3-b66e-441b-bbfe-1a77037e2fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:29:12.183992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.183969 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k" pod "08530db3-b66e-441b-bbfe-1a77037e2fd4" (UID: "08530db3-b66e-441b-bbfe-1a77037e2fd4"). InnerVolumeSpecName "kube-api-access-cv28k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:12.283387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.283363 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv28k\" (UniqueName: \"kubernetes.io/projected/08530db3-b66e-441b-bbfe-1a77037e2fd4-kube-api-access-cv28k\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:12.283387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.283386 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08530db3-b66e-441b-bbfe-1a77037e2fd4-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:12.840010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.839983 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" Oct 29 22:29:12.840214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.840018 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5" event={"ID":"08530db3-b66e-441b-bbfe-1a77037e2fd4","Type":"ContainerDied","Data":"665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1"} Oct 29 22:29:12.840214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:12.840065 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="665fb6d2d9278145d61c531a917090cd4776d2216463f7794dcbbdb2e3bdbcf1" Oct 29 22:29:23.478467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:23.478436 4543 scope.go:122] "RemoveContainer" containerID="60eea0253970e722721b6c2225dfda1d0c0b4190c91a4918a632bb1bb3c6f33c" Oct 29 22:29:24.442800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.442742 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:24.444942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.444804 4543 cpu_manager.go:414] "containerMap: removing container" podUID="08530db3-b66e-441b-bbfe-1a77037e2fd4" containerName="revoke-keys" Oct 29 22:29:24.444942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.444839 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08530db3-b66e-441b-bbfe-1a77037e2fd4" containerName="revoke-keys" Oct 29 22:29:24.445211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.445124 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="08530db3-b66e-441b-bbfe-1a77037e2fd4" containerName="revoke-keys" Oct 29 22:29:24.452880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.452857 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.466373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.466348 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:24.473117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.471796 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"agc-cred-89ed28687c44\"" Oct 29 22:29:24.473117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.471837 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-e2e--0fbd5c57-dockercfg-2nbzj\"" Oct 29 22:29:24.473117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.471939 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"kube-root-ca.crt\"" Oct 29 22:29:24.482540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.482517 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-50d4af\"" Oct 29 22:29:24.482540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.482529 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"openshift-service-ca.crt\"" Oct 29 22:29:24.482887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.482548 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-50d4af\"/\"routing-overlay\"" Oct 29 22:29:24.638099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.635565 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.638099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.635687 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6fv\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.638099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.635836 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.638099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.635917 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.741292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.741381 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.741667 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.741845 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6fv\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.743098 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.747098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.743831 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.799098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.789758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6fv\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:24.816665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:24.816628 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:25.070917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:25.070829 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:25.497099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:25.488711 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:26.037772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:26.037745 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" event={"ID":"8e79af91-c6c7-448e-a687-779b6eb0f90e","Type":"ContainerStarted","Data":"8a9d9f33f38ebb094f0f511a48a68845dc0aa15775456996444c8787364cf4e3"} Oct 29 22:29:27.048474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:27.048434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" event={"ID":"8e79af91-c6c7-448e-a687-779b6eb0f90e","Type":"ContainerStarted","Data":"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4"} Oct 29 22:29:27.048886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:27.048525 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:27.093701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:27.093656 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" podStartSLOduration=3.093642656 podStartE2EDuration="3.093642656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:27.092839048 +0000 UTC m=+2828.834769718" watchObservedRunningTime="2026-10-29 22:29:27.093642656 +0000 UTC m=+2828.835573328" Oct 29 22:29:27.239496 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.239460 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.239496 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.239489 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.239774 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.239550 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:27.739531909 +0000 UTC m=+2829.481462575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.744769 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.744733 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.744769 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.744760 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.744983 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.744823 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:28.744809874 +0000 UTC m=+2830.486740525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:27.962793 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:27.962756 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:29:28.763707 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:28.763667 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:28.763707 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:28.763693 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:28.764191 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:28.763748 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:30.763734065 +0000 UTC m=+2832.505664716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:30.792858 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:30.792767 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:30.792858 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:30.792799 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:30.793331 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:30.792864 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:34.792847764 +0000 UTC m=+2836.534778437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:34.884911 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:34.884871 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:34.884911 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:34.884906 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:34.885591 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:34.885026 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:42.885003997 +0000 UTC m=+2844.626934668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:38.060115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:38.060070 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:39.037253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:39.037221 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp"] Oct 29 22:29:39.050112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:39.050069 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e97b061-654pp"] Oct 29 22:29:39.131794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:39.131760 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:40.445176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:40.445138 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66131f37-c57d-438f-bd18-19537b1b41e7" path="/var/lib/kubelet/pods/66131f37-c57d-438f-bd18-19537b1b41e7/volumes" Oct 29 22:29:42.969764 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:42.969728 4543 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-50d4af/agc-cred-89ed28687c44: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:42.969764 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:42.969755 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n: secret "agc-cred-89ed28687c44" not found Oct 29 22:29:42.970304 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:42.969828 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials podName:8e79af91-c6c7-448e-a687-779b6eb0f90e nodeName:}" failed. No retries permitted until 2026-10-29 22:29:58.969810716 +0000 UTC m=+2860.711741386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e") : secret "agc-cred-89ed28687c44" not found Oct 29 22:29:43.014774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.014744 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:29:43.021510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.021491 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.038838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.038816 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:29:43.045120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.045055 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:29:43.045120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.045048 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-75ea0c-serving-cert\"" Oct 29 22:29:43.045311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.045120 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:29:43.045311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.045278 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:29:43.051558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.051539 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8hhr\"" Oct 29 22:29:43.071486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.071461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.071623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.071607 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.071763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.071746 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v57\" (UniqueName: \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.071839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.071828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.172916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.172873 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v57\" (UniqueName: \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.173146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.173001 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.173242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.173188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.173368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.173345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.173833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.173806 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.175682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.175655 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.175835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.175820 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.201550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.201529 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v57\" (UniqueName: \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\") pod \"maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:43.338449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:43.338380 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:44.132178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.132120 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" podUID="8e79af91-c6c7-448e-a687-779b6eb0f90e" containerName="payload-processing" containerID="cri-o://75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4" gracePeriod=25 Oct 29 22:29:44.155989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.155963 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:29:44.211054 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:29:44.211025 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0434f25f_8dde_4881_9a86_dbe7d952cb3a.slice/crio-b0499acb44c4fbedac4dd895acec51727f0a51234f09f039ffcb2790db60d986 WatchSource:0}: Error finding container b0499acb44c4fbedac4dd895acec51727f0a51234f09f039ffcb2790db60d986: Status 404 returned error can't find the container with id b0499acb44c4fbedac4dd895acec51727f0a51234f09f039ffcb2790db60d986 Oct 29 22:29:44.237282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.237261 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:29:44.455174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.455150 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:44.460454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.460435 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\") pod \"8e79af91-c6c7-448e-a687-779b6eb0f90e\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " Oct 29 22:29:44.460543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.460533 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\") pod \"8e79af91-c6c7-448e-a687-779b6eb0f90e\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " Oct 29 22:29:44.460766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.460742 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\") pod \"8e79af91-c6c7-448e-a687-779b6eb0f90e\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " Oct 29 22:29:44.460944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.460929 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\") pod \"8e79af91-c6c7-448e-a687-779b6eb0f90e\" (UID: \"8e79af91-c6c7-448e-a687-779b6eb0f90e\") " Oct 29 22:29:44.461220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.461200 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume" pod "8e79af91-c6c7-448e-a687-779b6eb0f90e" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:44.461514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.461486 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume" pod "8e79af91-c6c7-448e-a687-779b6eb0f90e" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:29:44.461829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.461812 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:44.461876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.461841 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e79af91-c6c7-448e-a687-779b6eb0f90e-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:44.462855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.462829 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials" pod "8e79af91-c6c7-448e-a687-779b6eb0f90e" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:44.462940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.462906 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv" pod "8e79af91-c6c7-448e-a687-779b6eb0f90e" (UID: "8e79af91-c6c7-448e-a687-779b6eb0f90e"). InnerVolumeSpecName "kube-api-access-ks6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:29:44.563746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.563713 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:44.563746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:44.563740 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks6fv\" (UniqueName: \"kubernetes.io/projected/8e79af91-c6c7-448e-a687-779b6eb0f90e-kube-api-access-ks6fv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:29:45.307165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.307062 4543 generic.go:356] "Generic (PLEG): container finished" podID="8e79af91-c6c7-448e-a687-779b6eb0f90e" containerID="75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4" exitCode=0 Oct 29 22:29:45.307832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.307795 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" event={"ID":"8e79af91-c6c7-448e-a687-779b6eb0f90e","Type":"ContainerDied","Data":"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4"} Oct 29 22:29:45.308163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.308143 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" event={"ID":"8e79af91-c6c7-448e-a687-779b6eb0f90e","Type":"ContainerDied","Data":"8a9d9f33f38ebb094f0f511a48a68845dc0aa15775456996444c8787364cf4e3"} Oct 29 22:29:45.308321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.308309 4543 scope.go:122] "RemoveContainer" containerID="75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4" Oct 29 22:29:45.308627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.308612 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n" Oct 29 22:29:45.312808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.312743 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" event={"ID":"0434f25f-8dde-4881-9a86-dbe7d952cb3a","Type":"ContainerStarted","Data":"b0499acb44c4fbedac4dd895acec51727f0a51234f09f039ffcb2790db60d986"} Oct 29 22:29:45.329755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.329670 4543 scope.go:122] "RemoveContainer" containerID="75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4" Oct 29 22:29:45.330301 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:45.330269 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4\": container with ID starting with 75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4 not found: ID does not exist" containerID="75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4" Oct 29 22:29:45.330430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.330307 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4"} err="failed to get container status \"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4\": rpc error: code = NotFound desc = could not find container \"75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4\": container with ID starting with 75141a5f22c99c88d3c49c63da8a670160083d4c7a5a11c2903aad9160ec03a4 not found: ID does not exist" Oct 29 22:29:45.372974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.372944 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:45.384898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:45.384871 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-50d4af/payload-processing-external-model-e2e-extmod-a-50d4af-6f9bg8z7n"] Oct 29 22:29:46.053604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.053562 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:29:46.054929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.054903 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8e79af91-c6c7-448e-a687-779b6eb0f90e" containerName="payload-processing" Oct 29 22:29:46.054929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.054926 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e79af91-c6c7-448e-a687-779b6eb0f90e" containerName="payload-processing" Oct 29 22:29:46.055290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.055273 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e79af91-c6c7-448e-a687-779b6eb0f90e" containerName="payload-processing" Oct 29 22:29:46.065438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.065414 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.082973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.082931 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:29:46.083674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.083651 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"agc-cred-158185d37305\"" Oct 29 22:29:46.083789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.083718 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"agc-cred-6cfe15a5e916\"" Oct 29 22:29:46.096418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.096026 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"openshift-service-ca.crt\"" Oct 29 22:29:46.096418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.096111 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-adfb9d\"" Oct 29 22:29:46.096418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.096141 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"routing-overlay\"" Oct 29 22:29:46.096418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.096216 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"kube-root-ca.crt\"" Oct 29 22:29:46.096418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.096297 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-e2e--b64eb102-dockercfg-2z6nl\"" Oct 29 22:29:46.177976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.177932 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.177976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.177981 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glrz\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.178251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.178121 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.178251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.178151 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.280233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.280177 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.280431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.280242 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glrz\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.280431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.280337 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.280431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.280380 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.280930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.280903 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.281253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.281226 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.283301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.283262 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.306646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.306594 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glrz\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.393111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.393062 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e79af91-c6c7-448e-a687-779b6eb0f90e" path="/var/lib/kubelet/pods/8e79af91-c6c7-448e-a687-779b6eb0f90e/volumes" Oct 29 22:29:46.407303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.407278 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:46.740476 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:29:46.740449 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b613a2_9318_44e9_9d74_ab7780385006.slice/crio-a4a3250ccad953ffee3ae394879ee6f75934bed4fa79b35f9c9f215769a98080 WatchSource:0}: Error finding container a4a3250ccad953ffee3ae394879ee6f75934bed4fa79b35f9c9f215769a98080: Status 404 returned error can't find the container with id a4a3250ccad953ffee3ae394879ee6f75934bed4fa79b35f9c9f215769a98080 Oct 29 22:29:46.742930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:46.742872 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:29:47.342342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:47.342304 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" event={"ID":"78b613a2-9318-44e9-9d74-ab7780385006","Type":"ContainerStarted","Data":"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918"} Oct 29 22:29:47.342546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:47.342373 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" event={"ID":"78b613a2-9318-44e9-9d74-ab7780385006","Type":"ContainerStarted","Data":"a4a3250ccad953ffee3ae394879ee6f75934bed4fa79b35f9c9f215769a98080"} Oct 29 22:29:47.342546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:47.342430 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:29:47.391258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:47.391211 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" podStartSLOduration=1.391196723 podStartE2EDuration="1.391196723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:47.387667088 +0000 UTC m=+2849.129597760" watchObservedRunningTime="2026-10-29 22:29:47.391196723 +0000 UTC m=+2849.133127396" Oct 29 22:29:48.353570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:48.353538 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" event={"ID":"0434f25f-8dde-4881-9a86-dbe7d952cb3a","Type":"ContainerStarted","Data":"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b"} Oct 29 22:29:48.353957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:48.353614 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:48.400746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:48.400703 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" podStartSLOduration=3.189246183 podStartE2EDuration="6.400688947s" totalImagesPullingTime="3.211442764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:48.398253655 +0000 UTC m=+2850.140184324" watchObservedRunningTime="2026-10-29 22:29:48.400688947 +0000 UTC m=+2850.142619619" Oct 29 22:29:52.055145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:52.050644 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr"] Oct 29 22:29:52.066423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:52.066390 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b974f38-f27dr"] Oct 29 22:29:52.391105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:52.391015 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e77e9dc-88d1-474c-840f-e278269f1099" path="/var/lib/kubelet/pods/2e77e9dc-88d1-474c-840f-e278269f1099/volumes" Oct 29 22:29:54.369603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:54.369573 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:29:57.644251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:57.644219 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:29:57.963134 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:29:57.963095 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:29:58.357003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:29:58.356937 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:30:02.645474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:02.645438 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerName="payload-processing" containerID="cri-o://30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05" gracePeriod=25 Oct 29 22:30:02.737509 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:02.737486 4543 logging.go:55] [core] [Channel #19362 SubChannel #19363]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.98:50052", ServerName: "10.133.1.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.98:50052: connect: connection refused" Oct 29 22:30:02.969586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:02.969552 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:30:02.999795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:02.999766 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\") pod \"8b221315-7dd4-4019-839b-b5d27d83fc36\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " Oct 29 22:30:02.999917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:02.999852 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\") pod \"8b221315-7dd4-4019-839b-b5d27d83fc36\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " Oct 29 22:30:03.000069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000045 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\") pod \"8b221315-7dd4-4019-839b-b5d27d83fc36\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " Oct 29 22:30:03.000218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000137 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume" pod "8b221315-7dd4-4019-839b-b5d27d83fc36" (UID: "8b221315-7dd4-4019-839b-b5d27d83fc36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:03.000218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000208 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume" pod "8b221315-7dd4-4019-839b-b5d27d83fc36" (UID: "8b221315-7dd4-4019-839b-b5d27d83fc36"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:03.000344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000330 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\") pod \"8b221315-7dd4-4019-839b-b5d27d83fc36\" (UID: \"8b221315-7dd4-4019-839b-b5d27d83fc36\") " Oct 29 22:30:03.000899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000843 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:03.000899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.000863 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b221315-7dd4-4019-839b-b5d27d83fc36-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:03.002460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.002441 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85" pod "8b221315-7dd4-4019-839b-b5d27d83fc36" (UID: "8b221315-7dd4-4019-839b-b5d27d83fc36"). InnerVolumeSpecName "kube-api-access-jvk85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:03.002549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.002467 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials" pod "8b221315-7dd4-4019-839b-b5d27d83fc36" (UID: "8b221315-7dd4-4019-839b-b5d27d83fc36"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:03.036205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.036178 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:30:03.037932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.037909 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerName="payload-processing" Oct 29 22:30:03.037999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.037937 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerName="payload-processing" Oct 29 22:30:03.038259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.038243 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerName="payload-processing" Oct 29 22:30:03.043686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.043659 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.058625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.058599 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:30:03.060717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.060676 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e7b56d\"/\"agc-cred-ba94f09e17ec\"" Oct 29 22:30:03.101778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.101749 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.101913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.101788 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzsb\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.101913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.101816 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.102058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.101938 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.102291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.102273 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:03.102364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.102293 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvk85\" (UniqueName: \"kubernetes.io/projected/8b221315-7dd4-4019-839b-b5d27d83fc36-kube-api-access-jvk85\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:03.204876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.204776 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.204876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.204837 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzsb\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.204876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.204875 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.205226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.204919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.205925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.205899 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.205925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.205911 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.208556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.208531 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.230835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.230806 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzsb\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\") pod \"payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.364122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.364098 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:03.514066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.514032 4543 generic.go:356] "Generic (PLEG): container finished" podID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerID="30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05" exitCode=0 Oct 29 22:30:03.514223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.514087 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" Oct 29 22:30:03.514223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.514133 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" event={"ID":"8b221315-7dd4-4019-839b-b5d27d83fc36","Type":"ContainerDied","Data":"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05"} Oct 29 22:30:03.514223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.514188 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" event={"ID":"8b221315-7dd4-4019-839b-b5d27d83fc36","Type":"ContainerDied","Data":"d4c3c3e985ce53a38d965231acdf4b9f0605b2e7a22308875f181adb5acd6148"} Oct 29 22:30:03.514223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.514217 4543 scope.go:122] "RemoveContainer" containerID="30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05" Oct 29 22:30:03.527441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.527419 4543 scope.go:122] "RemoveContainer" containerID="30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05" Oct 29 22:30:03.527729 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:03.527709 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05\": container with ID starting with 30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05 not found: ID does not exist" containerID="30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05" Oct 29 22:30:03.527796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.527736 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05"} err="failed to get container status \"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05\": rpc error: code = NotFound desc = could not find container \"30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05\": container with ID starting with 30f39c40ab088dd0ab7c173fbabef4e3fe144b54d22cc90fcb04338d62a8fb05 not found: ID does not exist" Oct 29 22:30:03.583151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.583120 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:30:03.590958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.590928 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq"] Oct 29 22:30:03.683217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.683191 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:30:03.683596 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:03.683534 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9222c51_1bb6_4197_8d07_a6099b432436.slice/crio-b461817f8478961777a74f942792e062ad8a8a807cb40c7647e78209d4085fd4 WatchSource:0}: Error finding container b461817f8478961777a74f942792e062ad8a8a807cb40c7647e78209d4085fd4: Status 404 returned error can't find the container with id b461817f8478961777a74f942792e062ad8a8a807cb40c7647e78209d4085fd4 Oct 29 22:30:03.737825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:03.737792 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5b68qhq" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.98:50052\" within 1s: context deadline exceeded" Oct 29 22:30:04.389558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:04.389525 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b221315-7dd4-4019-839b-b5d27d83fc36" path="/var/lib/kubelet/pods/8b221315-7dd4-4019-839b-b5d27d83fc36/volumes" Oct 29 22:30:04.524978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:04.524945 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" event={"ID":"c9222c51-1bb6-4197-8d07-a6099b432436","Type":"ContainerStarted","Data":"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7"} Oct 29 22:30:04.525202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:04.524997 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" event={"ID":"c9222c51-1bb6-4197-8d07-a6099b432436","Type":"ContainerStarted","Data":"b461817f8478961777a74f942792e062ad8a8a807cb40c7647e78209d4085fd4"} Oct 29 22:30:04.525202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:04.525024 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:04.570189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:04.570145 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" podStartSLOduration=8.570130563 podStartE2EDuration="8.570130563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:04.569413158 +0000 UTC m=+2866.311343831" watchObservedRunningTime="2026-10-29 22:30:04.570130563 +0000 UTC m=+2866.312061240" Oct 29 22:30:15.550714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:15.550674 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:30:15.637899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:15.637870 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:30:16.600337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:16.600298 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:30:20.638993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:20.638952 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" podUID="669bb78c-ec79-497b-a576-988b357e03cd" containerName="payload-processing" containerID="cri-o://d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d" gracePeriod=25 Oct 29 22:30:21.029603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.029578 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:30:21.091981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.091930 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:21.093783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.093757 4543 cpu_manager.go:414] "containerMap: removing container" podUID="669bb78c-ec79-497b-a576-988b357e03cd" containerName="payload-processing" Oct 29 22:30:21.093972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.093955 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669bb78c-ec79-497b-a576-988b357e03cd" containerName="payload-processing" Oct 29 22:30:21.094482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.094458 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="669bb78c-ec79-497b-a576-988b357e03cd" containerName="payload-processing" Oct 29 22:30:21.099727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.099703 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:21.100306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.100258 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.106941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.106915 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.110722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.110697 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:21.112805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.112774 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\") pod \"669bb78c-ec79-497b-a576-988b357e03cd\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " Oct 29 22:30:21.113140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.113064 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\") pod \"669bb78c-ec79-497b-a576-988b357e03cd\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " Oct 29 22:30:21.113273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.113174 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\") pod \"669bb78c-ec79-497b-a576-988b357e03cd\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " Oct 29 22:30:21.113273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.113209 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\") pod \"669bb78c-ec79-497b-a576-988b357e03cd\" (UID: \"669bb78c-ec79-497b-a576-988b357e03cd\") " Oct 29 22:30:21.113848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.113823 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume" pod "669bb78c-ec79-497b-a576-988b357e03cd" (UID: "669bb78c-ec79-497b-a576-988b357e03cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:21.114227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.114201 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.114742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.114713 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume" pod "669bb78c-ec79-497b-a576-988b357e03cd" (UID: "669bb78c-ec79-497b-a576-988b357e03cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:21.131208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.120783 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:30:21.131208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.126018 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p" pod "669bb78c-ec79-497b-a576-988b357e03cd" (UID: "669bb78c-ec79-497b-a576-988b357e03cd"). InnerVolumeSpecName "kube-api-access-gcj2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:21.132787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.131848 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials" pod "669bb78c-ec79-497b-a576-988b357e03cd" (UID: "669bb78c-ec79-497b-a576-988b357e03cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:21.132787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.131866 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:21.132787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.131951 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.138903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.138882 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:21.140756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.140735 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.147062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.147044 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:21.147338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.147279 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.151322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.151302 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:21.156713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.156694 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:21.161945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.161925 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:21.166621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.166601 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:21.215310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215283 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4hx\" (UniqueName: \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.215452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215401 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.215525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215494 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rfd\" (UniqueName: \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.215574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215561 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.215620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215607 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.215682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg5c\" (UniqueName: \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.215738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215708 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcj57\" (UniqueName: \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.215843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215823 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.215943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w76b\" (UniqueName: \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.216011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.215967 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.216213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.216197 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669bb78c-ec79-497b-a576-988b357e03cd-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.216305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.216217 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.216305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.216234 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcj2p\" (UniqueName: \"kubernetes.io/projected/669bb78c-ec79-497b-a576-988b357e03cd-kube-api-access-gcj2p\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:21.317081 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.317226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317135 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.317226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317165 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg5c\" (UniqueName: \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.317226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317198 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcj57\" (UniqueName: \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.317419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317273 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.317419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317386 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w76b\" (UniqueName: \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.317533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317422 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.317622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317605 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4hx\" (UniqueName: \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.317689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317677 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.317770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317757 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rfd\" (UniqueName: \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.317832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317815 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.317925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.317900 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.318061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.318045 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.318268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.318244 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.318423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.318407 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.344288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.344266 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4hx\" (UniqueName: \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.344403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.344359 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w76b\" (UniqueName: \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.344403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.344388 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcj57\" (UniqueName: \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.344692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.344675 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rfd\" (UniqueName: \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.344731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.344713 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg5c\" (UniqueName: \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.497940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.497871 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:21.509281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.509257 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:21.525657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.525624 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:21.532800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.532744 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:21.539947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.539922 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:21.608290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.608148 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" podUID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" containerName="payload-processing" containerID="cri-o://8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8" gracePeriod=25 Oct 29 22:30:21.693483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.692507 4543 generic.go:356] "Generic (PLEG): container finished" podID="669bb78c-ec79-497b-a576-988b357e03cd" containerID="d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d" exitCode=0 Oct 29 22:30:21.693483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.692758 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" event={"ID":"669bb78c-ec79-497b-a576-988b357e03cd","Type":"ContainerDied","Data":"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d"} Oct 29 22:30:21.693483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.692832 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" event={"ID":"669bb78c-ec79-497b-a576-988b357e03cd","Type":"ContainerDied","Data":"880d48622d1b09e16aea2ce9711f4ca4c8a599875abafb2e03c486f500a02a17"} Oct 29 22:30:21.693483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.692856 4543 scope.go:122] "RemoveContainer" containerID="d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d" Oct 29 22:30:21.693483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.692999 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms" Oct 29 22:30:21.736547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.736521 4543 scope.go:122] "RemoveContainer" containerID="d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d" Oct 29 22:30:21.736957 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:21.736931 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d\": container with ID starting with d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d not found: ID does not exist" containerID="d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d" Oct 29 22:30:21.737053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.736969 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d"} err="failed to get container status \"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d\": rpc error: code = NotFound desc = could not find container \"d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d\": container with ID starting with d51323d73ab30bc52d32144d627904ee5dc1e9894da6ab6b5a7dfdebd1165d2d not found: ID does not exist" Oct 29 22:30:21.797800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.797718 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:30:21.829284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.829257 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-5f54h46ms"] Oct 29 22:30:21.864558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:21.864489 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:21.867709 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:21.867680 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ffaa9e_56c1_4c3d_8441_4e38be2f089f.slice/crio-3cb59c33fc079523a43449c71b99590df8e7521100e1c74ab50c820a6cae3fc1 WatchSource:0}: Error finding container 3cb59c33fc079523a43449c71b99590df8e7521100e1c74ab50c820a6cae3fc1: Status 404 returned error can't find the container with id 3cb59c33fc079523a43449c71b99590df8e7521100e1c74ab50c820a6cae3fc1 Oct 29 22:30:22.025821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.025791 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:30:22.083700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.083606 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:22.087105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.087043 4543 cpu_manager.go:414] "containerMap: removing container" podUID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" containerName="payload-processing" Oct 29 22:30:22.087105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.087088 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" containerName="payload-processing" Oct 29 22:30:22.087377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.087363 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" containerName="payload-processing" Oct 29 22:30:22.127167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.112981 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:22.127167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.113157 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.127167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.120561 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:22.127167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.121237 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.132705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.132381 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:22.132705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.132623 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.139132 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:22.139044 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03055aa_841f_4232_8009_046f61222149.slice/crio-b1ecabac1409f060c46fccddaa6cb19438e63d0dd1563a65271683d288499f78 WatchSource:0}: Error finding container b1ecabac1409f060c46fccddaa6cb19438e63d0dd1563a65271683d288499f78: Status 404 returned error can't find the container with id b1ecabac1409f060c46fccddaa6cb19438e63d0dd1563a65271683d288499f78 Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.142211 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.142428 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\") pod \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.142557 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\") pod \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.142810 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\") pod \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.142895 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\") pod \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\" (UID: \"08f87e63-fb77-483d-9e4c-d55ef00e31e5\") " Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.143254 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.143308 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwww\" (UniqueName: \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.143358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.143536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z28m\" (UniqueName: \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.148099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.144766 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.149863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.149783 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7" pod "08f87e63-fb77-483d-9e4c-d55ef00e31e5" (UID: "08f87e63-fb77-483d-9e4c-d55ef00e31e5"). InnerVolumeSpecName "kube-api-access-pd9p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:22.152950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.150004 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials" pod "08f87e63-fb77-483d-9e4c-d55ef00e31e5" (UID: "08f87e63-fb77-483d-9e4c-d55ef00e31e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:22.152950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.150334 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume" pod "08f87e63-fb77-483d-9e4c-d55ef00e31e5" (UID: "08f87e63-fb77-483d-9e4c-d55ef00e31e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:22.152950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.150976 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:22.152950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.151006 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:22.163141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.151067 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.163141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.155036 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:22.164433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.163430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume" pod "08f87e63-fb77-483d-9e4c-d55ef00e31e5" (UID: "08f87e63-fb77-483d-9e4c-d55ef00e31e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:22.169426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.168683 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:22.189675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.188641 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:22.199003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.198980 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:22.206448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.206370 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:22.219821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.216324 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:22.224091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.224056 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:22.247161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247111 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.247307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.247307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247293 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.247435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247330 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrthz\" (UniqueName: \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.247435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247371 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwww\" (UniqueName: \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.247435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.247614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.247561 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.248526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.248498 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.248648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.248517 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.248891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.248589 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z28m\" (UniqueName: \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.248981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.248957 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjjb\" (UniqueName: \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.249247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.249218 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mlz\" (UniqueName: \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.249379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.249360 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd9p7\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-kube-api-access-pd9p7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.249466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.249385 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.249466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.249400 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08f87e63-fb77-483d-9e4c-d55ef00e31e5-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.249466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.249415 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f87e63-fb77-483d-9e4c-d55ef00e31e5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:22.272455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.272421 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z28m\" (UniqueName: \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.272455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.272455 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwww\" (UniqueName: \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.350645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.350590 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjjb\" (UniqueName: \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.350767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.350747 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mlz\" (UniqueName: \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.350833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.350817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.350890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.350866 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.350932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.350910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrthz\" (UniqueName: \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.351133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.351115 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.351521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.351498 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.351638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.351582 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.351785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.351758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.391372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.391338 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669bb78c-ec79-497b-a576-988b357e03cd" path="/var/lib/kubelet/pods/669bb78c-ec79-497b-a576-988b357e03cd/volumes" Oct 29 22:30:22.436250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.436228 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mlz\" (UniqueName: \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.436371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.436298 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjjb\" (UniqueName: \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.436371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.436298 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrthz\" (UniqueName: \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.495060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.495016 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:22.499204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.499182 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:22.508297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.508275 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:22.526754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.526729 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:22.538317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.538154 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:22.705645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.705615 4543 generic.go:356] "Generic (PLEG): container finished" podID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" containerID="8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8" exitCode=0 Oct 29 22:30:22.706062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.705684 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" Oct 29 22:30:22.706062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.705695 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" event={"ID":"08f87e63-fb77-483d-9e4c-d55ef00e31e5","Type":"ContainerDied","Data":"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8"} Oct 29 22:30:22.706062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.705776 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88" event={"ID":"08f87e63-fb77-483d-9e4c-d55ef00e31e5","Type":"ContainerDied","Data":"ef8e9030eac1b637e622307b0320fca7a4ad04705f2de2f256b2866a04ef1b94"} Oct 29 22:30:22.706062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.705801 4543 scope.go:122] "RemoveContainer" containerID="8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8" Oct 29 22:30:22.707720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.707693 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerStarted","Data":"b31d89c8cfda5f5ba8eef601b1b8bfbeac04f643179358974b283eba11869bd1"} Oct 29 22:30:22.709386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.709366 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerStarted","Data":"b1ecabac1409f060c46fccddaa6cb19438e63d0dd1563a65271683d288499f78"} Oct 29 22:30:22.710968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.710940 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerStarted","Data":"03ec6a280409c4d837a4576fe9a8e60aa7a14e772b24f873a33e58fac561ea0a"} Oct 29 22:30:22.713528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.713505 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerStarted","Data":"6fe5d73f630b2408242d8d43e55ec8b2205cef022f2677397e76686e2f587d39"} Oct 29 22:30:22.715052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.715031 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerStarted","Data":"9ca623b4338f922f3d7966e9f8c0a9dfe8097e4b245ba6ae50db012939d15e14"} Oct 29 22:30:22.715166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.715120 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerStarted","Data":"3cb59c33fc079523a43449c71b99590df8e7521100e1c74ab50c820a6cae3fc1"} Oct 29 22:30:22.786122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.786020 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" podStartSLOduration=22.786000051 podStartE2EDuration="22.786000051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:22.774366873 +0000 UTC m=+2884.516297546" watchObservedRunningTime="2026-10-29 22:30:22.786000051 +0000 UTC m=+2884.527930726" Oct 29 22:30:22.860114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.859307 4543 scope.go:122] "RemoveContainer" containerID="8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8" Oct 29 22:30:22.860114 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:22.859806 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8\": container with ID starting with 8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8 not found: ID does not exist" containerID="8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8" Oct 29 22:30:22.860114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.859842 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8"} err="failed to get container status \"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8\": rpc error: code = NotFound desc = could not find container \"8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8\": container with ID starting with 8606edb6fad1d7e93734ca3e2ec393faefd4eca442ec890d47644a35437600f8 not found: ID does not exist" Oct 29 22:30:22.888550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.888492 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:30:22.895873 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:22.895542 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe78a37f_554f_437b_a04d_ab5a9d250a12.slice/crio-0877b9cf1691a14c9d463edbf119fcedf136265001baefc216ba4faf2105b31f WatchSource:0}: Error finding container 0877b9cf1691a14c9d463edbf119fcedf136265001baefc216ba4faf2105b31f: Status 404 returned error can't find the container with id 0877b9cf1691a14c9d463edbf119fcedf136265001baefc216ba4faf2105b31f Oct 29 22:30:22.907337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.905664 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:22.913192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.913168 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dc2gs88"] Oct 29 22:30:22.927756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:22.927733 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:23.120136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.120057 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:23.125469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.125448 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:23.145296 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:23.145268 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b9973f_61e1_4a36_a5b0_9efd5e623d9b.slice/crio-dfad44f1f3f6e19c9f7d8b2fd29873ed20a168a24800463f00d9bb5f7b924c77 WatchSource:0}: Error finding container dfad44f1f3f6e19c9f7d8b2fd29873ed20a168a24800463f00d9bb5f7b924c77: Status 404 returned error can't find the container with id dfad44f1f3f6e19c9f7d8b2fd29873ed20a168a24800463f00d9bb5f7b924c77 Oct 29 22:30:23.226586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.226557 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:23.234011 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:23.231542 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1301585_df11_436f_98dd_1350c1beb43f.slice/crio-e38c1f75a8707eaef1170e0f73ae84d0c47030992c6f5f08dc286fee1cddebc2 WatchSource:0}: Error finding container e38c1f75a8707eaef1170e0f73ae84d0c47030992c6f5f08dc286fee1cddebc2: Status 404 returned error can't find the container with id e38c1f75a8707eaef1170e0f73ae84d0c47030992c6f5f08dc286fee1cddebc2 Oct 29 22:30:23.236057 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:30:23.236034 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93154a42_5f4b_40e7_bcb6_ed2c4d652144.slice/crio-66c39e18dd3571f459613246dc99b7eb2d64e3d9d4dded1f52200d5f681bc390 WatchSource:0}: Error finding container 66c39e18dd3571f459613246dc99b7eb2d64e3d9d4dded1f52200d5f681bc390: Status 404 returned error can't find the container with id 66c39e18dd3571f459613246dc99b7eb2d64e3d9d4dded1f52200d5f681bc390 Oct 29 22:30:23.726674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.726640 4543 generic.go:356] "Generic (PLEG): container finished" podID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerID="c3e8daebbc603cca1169eef483b17cc419c016eb0621d9e3f634602632bb58cb" exitCode=60 Oct 29 22:30:23.727047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.726685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerDied","Data":"c3e8daebbc603cca1169eef483b17cc419c016eb0621d9e3f634602632bb58cb"} Oct 29 22:30:23.727047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.727007 4543 scope.go:122] "RemoveContainer" containerID="c3e8daebbc603cca1169eef483b17cc419c016eb0621d9e3f634602632bb58cb" Oct 29 22:30:23.728977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.728949 4543 generic.go:356] "Generic (PLEG): container finished" podID="f03055aa-841f-4232-8009-046f61222149" containerID="64a5edb7e4cda0938ca1de0bf5484c9a2b2a09322d8680bc7b9f4161607f48b9" exitCode=60 Oct 29 22:30:23.729446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.729045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerDied","Data":"64a5edb7e4cda0938ca1de0bf5484c9a2b2a09322d8680bc7b9f4161607f48b9"} Oct 29 22:30:23.729446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.729375 4543 scope.go:122] "RemoveContainer" containerID="64a5edb7e4cda0938ca1de0bf5484c9a2b2a09322d8680bc7b9f4161607f48b9" Oct 29 22:30:23.731898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.731876 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerStarted","Data":"66c39e18dd3571f459613246dc99b7eb2d64e3d9d4dded1f52200d5f681bc390"} Oct 29 22:30:23.734984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.734938 4543 generic.go:356] "Generic (PLEG): container finished" podID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerID="0cde28397e68b7a51e1cf6d7b8f8285844f531b45fd2fcc45e9422131002575e" exitCode=60 Oct 29 22:30:23.735203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.735012 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerDied","Data":"0cde28397e68b7a51e1cf6d7b8f8285844f531b45fd2fcc45e9422131002575e"} Oct 29 22:30:23.735353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.735332 4543 scope.go:122] "RemoveContainer" containerID="0cde28397e68b7a51e1cf6d7b8f8285844f531b45fd2fcc45e9422131002575e" Oct 29 22:30:23.741252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.741215 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerStarted","Data":"dfad44f1f3f6e19c9f7d8b2fd29873ed20a168a24800463f00d9bb5f7b924c77"} Oct 29 22:30:23.743429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.743404 4543 generic.go:356] "Generic (PLEG): container finished" podID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerID="473323547bfac8d632d75cddf8ef7da282e6b82c028a55efcf05994a78915379" exitCode=60 Oct 29 22:30:23.743531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.743497 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerDied","Data":"473323547bfac8d632d75cddf8ef7da282e6b82c028a55efcf05994a78915379"} Oct 29 22:30:23.743787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.743701 4543 scope.go:122] "RemoveContainer" containerID="473323547bfac8d632d75cddf8ef7da282e6b82c028a55efcf05994a78915379" Oct 29 22:30:23.748410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.748388 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerStarted","Data":"0b52dcd63fc77540639eeab54eb04a9004f167c77466e7cd8af8295017fbbd88"} Oct 29 22:30:23.748489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.748426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerStarted","Data":"6adf933959e5affb978ac8cd812775ffe58f55eb031fec5556b5bab6d0873869"} Oct 29 22:30:23.750371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.750352 4543 generic.go:356] "Generic (PLEG): container finished" podID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerID="9ca623b4338f922f3d7966e9f8c0a9dfe8097e4b245ba6ae50db012939d15e14" exitCode=60 Oct 29 22:30:23.750471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.750446 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerDied","Data":"9ca623b4338f922f3d7966e9f8c0a9dfe8097e4b245ba6ae50db012939d15e14"} Oct 29 22:30:23.750675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.750656 4543 scope.go:122] "RemoveContainer" containerID="9ca623b4338f922f3d7966e9f8c0a9dfe8097e4b245ba6ae50db012939d15e14" Oct 29 22:30:23.752323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.752300 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerStarted","Data":"0877b9cf1691a14c9d463edbf119fcedf136265001baefc216ba4faf2105b31f"} Oct 29 22:30:23.753848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.753830 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerStarted","Data":"e38c1f75a8707eaef1170e0f73ae84d0c47030992c6f5f08dc286fee1cddebc2"} Oct 29 22:30:23.979068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:23.979016 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" podStartSLOduration=23.979002821999998 podStartE2EDuration="23.979002822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:23.976441079 +0000 UTC m=+2885.718371752" watchObservedRunningTime="2026-10-29 22:30:23.979002822 +0000 UTC m=+2885.720933494" Oct 29 22:30:24.400415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.400376 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f87e63-fb77-483d-9e4c-d55ef00e31e5" path="/var/lib/kubelet/pods/08f87e63-fb77-483d-9e4c-d55ef00e31e5/volumes" Oct 29 22:30:24.769224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.769191 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerStarted","Data":"b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1"} Oct 29 22:30:24.771509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.771481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerStarted","Data":"bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd"} Oct 29 22:30:24.773503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.773482 4543 generic.go:356] "Generic (PLEG): container finished" podID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerID="c2902db26d50a082ca7897ca46c8f791342a304bb6fe3d15e47c509aaf52273b" exitCode=60 Oct 29 22:30:24.773599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.773548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerDied","Data":"c2902db26d50a082ca7897ca46c8f791342a304bb6fe3d15e47c509aaf52273b"} Oct 29 22:30:24.773777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.773762 4543 scope.go:122] "RemoveContainer" containerID="c2902db26d50a082ca7897ca46c8f791342a304bb6fe3d15e47c509aaf52273b" Oct 29 22:30:24.775882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.775803 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerStarted","Data":"154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b"} Oct 29 22:30:24.777674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.777652 4543 generic.go:356] "Generic (PLEG): container finished" podID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerID="98c410ef71c5bddaa69e81a059df993fb5872800146bd7f259d6d47494e03dd3" exitCode=60 Oct 29 22:30:24.777759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.777730 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerDied","Data":"98c410ef71c5bddaa69e81a059df993fb5872800146bd7f259d6d47494e03dd3"} Oct 29 22:30:24.777953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.777934 4543 scope.go:122] "RemoveContainer" containerID="98c410ef71c5bddaa69e81a059df993fb5872800146bd7f259d6d47494e03dd3" Oct 29 22:30:24.781446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.781427 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerStarted","Data":"60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26"} Oct 29 22:30:24.783214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.783192 4543 generic.go:356] "Generic (PLEG): container finished" podID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerID="0b52dcd63fc77540639eeab54eb04a9004f167c77466e7cd8af8295017fbbd88" exitCode=60 Oct 29 22:30:24.783302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.783263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerDied","Data":"0b52dcd63fc77540639eeab54eb04a9004f167c77466e7cd8af8295017fbbd88"} Oct 29 22:30:24.783456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.783441 4543 scope.go:122] "RemoveContainer" containerID="0b52dcd63fc77540639eeab54eb04a9004f167c77466e7cd8af8295017fbbd88" Oct 29 22:30:24.785433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.785413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerStarted","Data":"410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe"} Oct 29 22:30:24.787226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.787205 4543 generic.go:356] "Generic (PLEG): container finished" podID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerID="780796edd2c8efcbdd71b81f0c232e8a6ad225e63afa1815f8596d4f7ea666d7" exitCode=60 Oct 29 22:30:24.787309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.787280 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerDied","Data":"780796edd2c8efcbdd71b81f0c232e8a6ad225e63afa1815f8596d4f7ea666d7"} Oct 29 22:30:24.787549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.787523 4543 scope.go:122] "RemoveContainer" containerID="780796edd2c8efcbdd71b81f0c232e8a6ad225e63afa1815f8596d4f7ea666d7" Oct 29 22:30:24.789366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.789343 4543 generic.go:356] "Generic (PLEG): container finished" podID="d1301585-df11-436f-98dd-1350c1beb43f" containerID="295177fee14f8cc9909887799363b396feabaced199f15f8d80a90a9e020ea5b" exitCode=60 Oct 29 22:30:24.789366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.789422 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerDied","Data":"295177fee14f8cc9909887799363b396feabaced199f15f8d80a90a9e020ea5b"} Oct 29 22:30:24.789780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.789763 4543 scope.go:122] "RemoveContainer" containerID="295177fee14f8cc9909887799363b396feabaced199f15f8d80a90a9e020ea5b" Oct 29 22:30:24.818606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.818529 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" podStartSLOduration=24.818511654 podStartE2EDuration="24.818511654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:24.814250804 +0000 UTC m=+2886.556181480" watchObservedRunningTime="2026-10-29 22:30:24.818511654 +0000 UTC m=+2886.560442328" Oct 29 22:30:24.940857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:24.940794 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" podStartSLOduration=24.940782005 podStartE2EDuration="24.940782005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:24.939733685 +0000 UTC m=+2886.681664359" watchObservedRunningTime="2026-10-29 22:30:24.940782005 +0000 UTC m=+2886.682712677" Oct 29 22:30:25.067064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.067009 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" podStartSLOduration=25.066993666 podStartE2EDuration="25.066993666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.064662058 +0000 UTC m=+2886.806592745" watchObservedRunningTime="2026-10-29 22:30:25.066993666 +0000 UTC m=+2886.808924338" Oct 29 22:30:25.108301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.108257 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" podStartSLOduration=25.108243697 podStartE2EDuration="25.108243697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.105551863 +0000 UTC m=+2886.847482548" watchObservedRunningTime="2026-10-29 22:30:25.108243697 +0000 UTC m=+2886.850174369" Oct 29 22:30:25.801514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.801485 4543 generic.go:356] "Generic (PLEG): container finished" podID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerID="b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1" exitCode=60 Oct 29 22:30:25.801915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.801568 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerDied","Data":"b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1"} Oct 29 22:30:25.801915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.801644 4543 scope.go:122] "RemoveContainer" containerID="c3e8daebbc603cca1169eef483b17cc419c016eb0621d9e3f634602632bb58cb" Oct 29 22:30:25.801915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.801825 4543 scope.go:122] "RemoveContainer" containerID="b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1" Oct 29 22:30:25.802251 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:25.802217 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx_odh-ai-gateway-infra(45274a01-b73e-490b-9b0c-1529c86e0f17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" Oct 29 22:30:25.804000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.803976 4543 generic.go:356] "Generic (PLEG): container finished" podID="f03055aa-841f-4232-8009-046f61222149" containerID="bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd" exitCode=60 Oct 29 22:30:25.804172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.804041 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerDied","Data":"bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd"} Oct 29 22:30:25.804416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.804397 4543 scope.go:122] "RemoveContainer" containerID="bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd" Oct 29 22:30:25.804745 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:25.804725 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z_odh-ai-gateway-infra(f03055aa-841f-4232-8009-046f61222149)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" podUID="f03055aa-841f-4232-8009-046f61222149" Oct 29 22:30:25.806781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.806761 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerStarted","Data":"6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321"} Oct 29 22:30:25.809118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.809069 4543 generic.go:356] "Generic (PLEG): container finished" podID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerID="154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b" exitCode=60 Oct 29 22:30:25.809218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.809121 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerDied","Data":"154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b"} Oct 29 22:30:25.809418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.809399 4543 scope.go:122] "RemoveContainer" containerID="154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b" Oct 29 22:30:25.809752 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:25.809727 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l_odh-ai-gateway-infra(55e0645d-eeb1-44d2-a6fb-60c570f2dadc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" Oct 29 22:30:25.818524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.818499 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerStarted","Data":"fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951"} Oct 29 22:30:25.820834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.820800 4543 generic.go:356] "Generic (PLEG): container finished" podID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerID="60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26" exitCode=60 Oct 29 22:30:25.820934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.820870 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerDied","Data":"60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26"} Oct 29 22:30:25.821184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.821169 4543 scope.go:122] "RemoveContainer" containerID="60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26" Oct 29 22:30:25.821454 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:25.821428 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l_odh-ai-gateway-infra(3a126320-b649-4e72-bd62-8a6a4f4f9b2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" Oct 29 22:30:25.821987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.821967 4543 scope.go:122] "RemoveContainer" containerID="64a5edb7e4cda0938ca1de0bf5484c9a2b2a09322d8680bc7b9f4161607f48b9" Oct 29 22:30:25.824286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.824262 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerStarted","Data":"342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20"} Oct 29 22:30:25.826132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.826114 4543 generic.go:356] "Generic (PLEG): container finished" podID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerID="410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe" exitCode=60 Oct 29 22:30:25.826239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.826144 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerDied","Data":"410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe"} Oct 29 22:30:25.826492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.826472 4543 scope.go:122] "RemoveContainer" containerID="410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe" Oct 29 22:30:25.826746 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:25.826727 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn_odh-ai-gateway-infra(01ffaa9e-56c1-4c3d-8441-4e38be2f089f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" Oct 29 22:30:25.828503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.828478 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerStarted","Data":"a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96"} Oct 29 22:30:25.834921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.834900 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerStarted","Data":"e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7"} Oct 29 22:30:25.839369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.839352 4543 scope.go:122] "RemoveContainer" containerID="0cde28397e68b7a51e1cf6d7b8f8285844f531b45fd2fcc45e9422131002575e" Oct 29 22:30:25.855153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.855137 4543 scope.go:122] "RemoveContainer" containerID="473323547bfac8d632d75cddf8ef7da282e6b82c028a55efcf05994a78915379" Oct 29 22:30:25.870309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.870295 4543 scope.go:122] "RemoveContainer" containerID="9ca623b4338f922f3d7966e9f8c0a9dfe8097e4b245ba6ae50db012939d15e14" Oct 29 22:30:25.928613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.928560 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" podStartSLOduration=25.928540466 podStartE2EDuration="25.928540466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.923996817 +0000 UTC m=+2887.665927495" watchObservedRunningTime="2026-10-29 22:30:25.928540466 +0000 UTC m=+2887.670471142" Oct 29 22:30:25.970525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:25.970480 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" podStartSLOduration=25.970467978 podStartE2EDuration="25.970467978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:25.96517968 +0000 UTC m=+2887.707110352" watchObservedRunningTime="2026-10-29 22:30:25.970467978 +0000 UTC m=+2887.712398651" Oct 29 22:30:26.009288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.009211 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" podStartSLOduration=26.009198063 podStartE2EDuration="26.009198063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:26.005184544 +0000 UTC m=+2887.747115219" watchObservedRunningTime="2026-10-29 22:30:26.009198063 +0000 UTC m=+2887.751128735" Oct 29 22:30:26.051720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.051670 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" podStartSLOduration=26.051641075 podStartE2EDuration="26.051641075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:26.045686923 +0000 UTC m=+2887.787617596" watchObservedRunningTime="2026-10-29 22:30:26.051641075 +0000 UTC m=+2887.793571787" Oct 29 22:30:26.846478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.846449 4543 scope.go:122] "RemoveContainer" containerID="b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1" Oct 29 22:30:26.846944 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.846755 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx_odh-ai-gateway-infra(45274a01-b73e-490b-9b0c-1529c86e0f17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" Oct 29 22:30:26.848369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.848349 4543 scope.go:122] "RemoveContainer" containerID="bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd" Oct 29 22:30:26.848631 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.848611 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z_odh-ai-gateway-infra(f03055aa-841f-4232-8009-046f61222149)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" podUID="f03055aa-841f-4232-8009-046f61222149" Oct 29 22:30:26.850406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.850380 4543 generic.go:356] "Generic (PLEG): container finished" podID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerID="6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321" exitCode=60 Oct 29 22:30:26.850495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.850465 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerDied","Data":"6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321"} Oct 29 22:30:26.850557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.850537 4543 scope.go:122] "RemoveContainer" containerID="c2902db26d50a082ca7897ca46c8f791342a304bb6fe3d15e47c509aaf52273b" Oct 29 22:30:26.850675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.850659 4543 scope.go:122] "RemoveContainer" containerID="6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321" Oct 29 22:30:26.850976 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.850952 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m_odh-ai-gateway-infra(93154a42-5f4b-40e7-bcb6-ed2c4d652144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" Oct 29 22:30:26.852806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.852786 4543 scope.go:122] "RemoveContainer" containerID="154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b" Oct 29 22:30:26.853154 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.853115 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l_odh-ai-gateway-infra(55e0645d-eeb1-44d2-a6fb-60c570f2dadc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" Oct 29 22:30:26.854808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.854787 4543 generic.go:356] "Generic (PLEG): container finished" podID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerID="fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951" exitCode=60 Oct 29 22:30:26.854933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.854909 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerDied","Data":"fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951"} Oct 29 22:30:26.855190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.855169 4543 scope.go:122] "RemoveContainer" containerID="fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951" Oct 29 22:30:26.855472 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.855446 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9_odh-ai-gateway-infra(a5b9973f-61e1-4a36-a5b0-9efd5e623d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" Oct 29 22:30:26.857461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.857434 4543 scope.go:122] "RemoveContainer" containerID="60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26" Oct 29 22:30:26.857723 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.857697 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l_odh-ai-gateway-infra(3a126320-b649-4e72-bd62-8a6a4f4f9b2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" Oct 29 22:30:26.859657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.859634 4543 generic.go:356] "Generic (PLEG): container finished" podID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerID="342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20" exitCode=60 Oct 29 22:30:26.859766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.859657 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerDied","Data":"342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20"} Oct 29 22:30:26.860021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.859994 4543 scope.go:122] "RemoveContainer" containerID="342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20" Oct 29 22:30:26.860310 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.860289 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw_odh-ai-gateway-infra(ef7cfdce-886f-47c2-a060-229f7b51c338)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" Oct 29 22:30:26.862272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.862254 4543 scope.go:122] "RemoveContainer" containerID="410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe" Oct 29 22:30:26.862573 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.862548 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn_odh-ai-gateway-infra(01ffaa9e-56c1-4c3d-8441-4e38be2f089f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" Oct 29 22:30:26.864196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.864175 4543 generic.go:356] "Generic (PLEG): container finished" podID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerID="a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96" exitCode=60 Oct 29 22:30:26.864545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.864255 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerDied","Data":"a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96"} Oct 29 22:30:26.864652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.864454 4543 scope.go:122] "RemoveContainer" containerID="a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96" Oct 29 22:30:26.864894 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.864874 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc_odh-ai-gateway-infra(fe78a37f-554f-437b-a04d-ab5a9d250a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" Oct 29 22:30:26.866286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.866262 4543 generic.go:356] "Generic (PLEG): container finished" podID="d1301585-df11-436f-98dd-1350c1beb43f" containerID="e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7" exitCode=60 Oct 29 22:30:26.866385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.866337 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerDied","Data":"e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7"} Oct 29 22:30:26.866533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.866521 4543 scope.go:122] "RemoveContainer" containerID="e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7" Oct 29 22:30:26.866751 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:26.866736 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz_odh-ai-gateway-infra(d1301585-df11-436f-98dd-1350c1beb43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" podUID="d1301585-df11-436f-98dd-1350c1beb43f" Oct 29 22:30:26.869876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.869858 4543 scope.go:122] "RemoveContainer" containerID="98c410ef71c5bddaa69e81a059df993fb5872800146bd7f259d6d47494e03dd3" Oct 29 22:30:26.891093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.891049 4543 scope.go:122] "RemoveContainer" containerID="0b52dcd63fc77540639eeab54eb04a9004f167c77466e7cd8af8295017fbbd88" Oct 29 22:30:26.906235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.906216 4543 scope.go:122] "RemoveContainer" containerID="780796edd2c8efcbdd71b81f0c232e8a6ad225e63afa1815f8596d4f7ea666d7" Oct 29 22:30:26.921163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:26.921145 4543 scope.go:122] "RemoveContainer" containerID="295177fee14f8cc9909887799363b396feabaced199f15f8d80a90a9e020ea5b" Oct 29 22:30:27.896796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:27.896764 4543 scope.go:122] "RemoveContainer" containerID="6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321" Oct 29 22:30:27.897355 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.897325 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m_odh-ai-gateway-infra(93154a42-5f4b-40e7-bcb6-ed2c4d652144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" Oct 29 22:30:27.898862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:27.898833 4543 scope.go:122] "RemoveContainer" containerID="fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951" Oct 29 22:30:27.899404 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.899376 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9_odh-ai-gateway-infra(a5b9973f-61e1-4a36-a5b0-9efd5e623d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" Oct 29 22:30:27.901249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:27.901230 4543 scope.go:122] "RemoveContainer" containerID="342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20" Oct 29 22:30:27.901691 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.901658 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw_odh-ai-gateway-infra(ef7cfdce-886f-47c2-a060-229f7b51c338)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" Oct 29 22:30:27.903227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:27.903205 4543 scope.go:122] "RemoveContainer" containerID="a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96" Oct 29 22:30:27.903538 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.903515 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc_odh-ai-gateway-infra(fe78a37f-554f-437b-a04d-ab5a9d250a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" Oct 29 22:30:27.904937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:27.904918 4543 scope.go:122] "RemoveContainer" containerID="e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7" Oct 29 22:30:27.905278 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.905247 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz_odh-ai-gateway-infra(d1301585-df11-436f-98dd-1350c1beb43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" podUID="d1301585-df11-436f-98dd-1350c1beb43f" Oct 29 22:30:27.967734 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:27.967702 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:30:35.378581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:35.378550 4543 scope.go:122] "RemoveContainer" containerID="b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1" Oct 29 22:30:36.415152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:36.412922 4543 scope.go:122] "RemoveContainer" containerID="154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b" Oct 29 22:30:37.179922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.179836 4543 generic.go:356] "Generic (PLEG): container finished" podID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerID="2f5fd7e474cb71a47c1439cc7dd89866b08a09bafbc9806cb525c41a7ce9bdbc" exitCode=60 Oct 29 22:30:37.179922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.179892 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerDied","Data":"2f5fd7e474cb71a47c1439cc7dd89866b08a09bafbc9806cb525c41a7ce9bdbc"} Oct 29 22:30:37.180208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.179944 4543 scope.go:122] "RemoveContainer" containerID="b9bb53dfbb269c667bb360206e9d2227fac0db65baf3ad3999403907e73f1da1" Oct 29 22:30:37.180679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.180652 4543 scope.go:122] "RemoveContainer" containerID="2f5fd7e474cb71a47c1439cc7dd89866b08a09bafbc9806cb525c41a7ce9bdbc" Oct 29 22:30:37.183718 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:37.181798 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx_odh-ai-gateway-infra(45274a01-b73e-490b-9b0c-1529c86e0f17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" Oct 29 22:30:37.185436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.185403 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerStarted","Data":"291dcd428bd61cb279454766d6a30cfc4f07d17522aee642446faa08fd0e91e4"} Oct 29 22:30:37.378783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:37.378756 4543 scope.go:122] "RemoveContainer" containerID="6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321" Oct 29 22:30:38.286765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.286731 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:38.290585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.290548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerStarted","Data":"b0668999607c6692e0259b0e3185acc337444d04bc6a13940bb142618c7262ea"} Oct 29 22:30:38.294236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.294199 4543 generic.go:356] "Generic (PLEG): container finished" podID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerID="291dcd428bd61cb279454766d6a30cfc4f07d17522aee642446faa08fd0e91e4" exitCode=60 Oct 29 22:30:38.294439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.294413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerDied","Data":"291dcd428bd61cb279454766d6a30cfc4f07d17522aee642446faa08fd0e91e4"} Oct 29 22:30:38.294526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.294496 4543 scope.go:122] "RemoveContainer" containerID="154c2d4faa83c903c489698bda012c3f55ecc3adc8fd3ceb5b01933a48c1767b" Oct 29 22:30:38.295402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.295377 4543 scope.go:122] "RemoveContainer" containerID="291dcd428bd61cb279454766d6a30cfc4f07d17522aee642446faa08fd0e91e4" Oct 29 22:30:38.297230 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:38.297200 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l_odh-ai-gateway-infra(55e0645d-eeb1-44d2-a6fb-60c570f2dadc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" Oct 29 22:30:38.311647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.311620 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:38.395619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.395556 4543 scope.go:122] "RemoveContainer" containerID="410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe" Oct 29 22:30:38.396738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.396606 4543 scope.go:122] "RemoveContainer" containerID="e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7" Oct 29 22:30:38.397448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.397420 4543 scope.go:122] "RemoveContainer" containerID="a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96" Oct 29 22:30:38.398181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.398041 4543 scope.go:122] "RemoveContainer" containerID="bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd" Oct 29 22:30:38.398650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.398612 4543 scope.go:122] "RemoveContainer" containerID="fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951" Oct 29 22:30:38.615509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.615489 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:38.782375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.781608 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\") pod \"45274a01-b73e-490b-9b0c-1529c86e0f17\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " Oct 29 22:30:38.782375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.781768 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\" (UniqueName: \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\") pod \"45274a01-b73e-490b-9b0c-1529c86e0f17\" (UID: \"45274a01-b73e-490b-9b0c-1529c86e0f17\") " Oct 29 22:30:38.782375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.782062 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca" pod "45274a01-b73e-490b-9b0c-1529c86e0f17" (UID: "45274a01-b73e-490b-9b0c-1529c86e0f17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:38.782996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.782972 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45274a01-b73e-490b-9b0c-1529c86e0f17-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:38.784239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.784206 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57" pod "45274a01-b73e-490b-9b0c-1529c86e0f17" (UID: "45274a01-b73e-490b-9b0c-1529c86e0f17"). InnerVolumeSpecName "kube-api-access-hcj57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:38.885525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:38.885494 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcj57\" (UniqueName: \"kubernetes.io/projected/45274a01-b73e-490b-9b0c-1529c86e0f17-kube-api-access-hcj57\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:39.312111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.310791 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" Oct 29 22:30:39.312111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.310787 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx" event={"ID":"45274a01-b73e-490b-9b0c-1529c86e0f17","Type":"ContainerDied","Data":"b31d89c8cfda5f5ba8eef601b1b8bfbeac04f643179358974b283eba11869bd1"} Oct 29 22:30:39.312111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.310929 4543 scope.go:122] "RemoveContainer" containerID="2f5fd7e474cb71a47c1439cc7dd89866b08a09bafbc9806cb525c41a7ce9bdbc" Oct 29 22:30:39.313606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.313577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerStarted","Data":"4e5585f8141a6e1e692994666d8116a8550659dae4470ccd60c668008a8f58ea"} Oct 29 22:30:39.315908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.315883 4543 generic.go:356] "Generic (PLEG): container finished" podID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerID="b0668999607c6692e0259b0e3185acc337444d04bc6a13940bb142618c7262ea" exitCode=60 Oct 29 22:30:39.315908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.315899 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerDied","Data":"b0668999607c6692e0259b0e3185acc337444d04bc6a13940bb142618c7262ea"} Oct 29 22:30:39.316303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.316284 4543 scope.go:122] "RemoveContainer" containerID="b0668999607c6692e0259b0e3185acc337444d04bc6a13940bb142618c7262ea" Oct 29 22:30:39.316681 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:39.316652 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m_odh-ai-gateway-infra(93154a42-5f4b-40e7-bcb6-ed2c4d652144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" Oct 29 22:30:39.320359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.320329 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerStarted","Data":"149795c5e984be21019434a57224e52e0631b0745277d675a173f6b67841d8a2"} Oct 29 22:30:39.323116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.323060 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerStarted","Data":"ccc4efa290065f8983614920f71610ceee38541a97792765875c34e1d5113c38"} Oct 29 22:30:39.325223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.325200 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerStarted","Data":"ef4e8308013025b5e9bd7fd55eb70920200ddcce9b4ede8c4527d4064c4f4e6f"} Oct 29 22:30:39.356349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.326632 4543 scope.go:122] "RemoveContainer" containerID="6dc9d422d1bd31909272e0fe442656e699b304c45adf0b2987c9b05590856321" Oct 29 22:30:39.356349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.327502 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerStarted","Data":"a511c4dfb382f979cee06302bcbb34d4e12850173f191d5b53e4cfab03be9322"} Oct 29 22:30:39.379599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.379568 4543 scope.go:122] "RemoveContainer" containerID="60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26" Oct 29 22:30:39.421650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.419841 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:39.571741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.571708 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:39.578210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.578182 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888550-dwmjx"] Oct 29 22:30:39.627731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.627708 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:39.804990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.804959 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\" (UniqueName: \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\") pod \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " Oct 29 22:30:39.805610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.805583 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\") pod \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\" (UID: \"55e0645d-eeb1-44d2-a6fb-60c570f2dadc\") " Oct 29 22:30:39.806856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.806833 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca" pod "55e0645d-eeb1-44d2-a6fb-60c570f2dadc" (UID: "55e0645d-eeb1-44d2-a6fb-60c570f2dadc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:39.810353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.810330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx" pod "55e0645d-eeb1-44d2-a6fb-60c570f2dadc" (UID: "55e0645d-eeb1-44d2-a6fb-60c570f2dadc"). InnerVolumeSpecName "kube-api-access-zl4hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:39.889784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.889753 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:30:39.889965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.889944 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" containerID="cri-o://439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4" gracePeriod=30 Oct 29 22:30:39.908398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.908359 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:39.908398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:39.908397 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4hx\" (UniqueName: \"kubernetes.io/projected/55e0645d-eeb1-44d2-a6fb-60c570f2dadc-kube-api-access-zl4hx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:40.364558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.364438 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerStarted","Data":"b2deb09ad7342528a9a75c272d0cba4cd3e69f50274fb0b15fe66f740779b4ad"} Oct 29 22:30:40.367235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.367207 4543 generic.go:356] "Generic (PLEG): container finished" podID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerID="ccc4efa290065f8983614920f71610ceee38541a97792765875c34e1d5113c38" exitCode=60 Oct 29 22:30:40.367388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.367276 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerDied","Data":"ccc4efa290065f8983614920f71610ceee38541a97792765875c34e1d5113c38"} Oct 29 22:30:40.367388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.367333 4543 scope.go:122] "RemoveContainer" containerID="410d7f9abbb20e7f6c8d124d84f7bd76e348a738e49a13b3c9c4813e44b52cfe" Oct 29 22:30:40.367721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.367699 4543 scope.go:122] "RemoveContainer" containerID="ccc4efa290065f8983614920f71610ceee38541a97792765875c34e1d5113c38" Oct 29 22:30:40.368240 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:40.368201 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn_odh-ai-gateway-infra(01ffaa9e-56c1-4c3d-8441-4e38be2f089f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" Oct 29 22:30:40.370107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.370063 4543 generic.go:356] "Generic (PLEG): container finished" podID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerID="ef4e8308013025b5e9bd7fd55eb70920200ddcce9b4ede8c4527d4064c4f4e6f" exitCode=60 Oct 29 22:30:40.370585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.370278 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerDied","Data":"ef4e8308013025b5e9bd7fd55eb70920200ddcce9b4ede8c4527d4064c4f4e6f"} Oct 29 22:30:40.371187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.371155 4543 scope.go:122] "RemoveContainer" containerID="ef4e8308013025b5e9bd7fd55eb70920200ddcce9b4ede8c4527d4064c4f4e6f" Oct 29 22:30:40.371871 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:40.371534 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc_odh-ai-gateway-infra(fe78a37f-554f-437b-a04d-ab5a9d250a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" Oct 29 22:30:40.373567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.373543 4543 generic.go:356] "Generic (PLEG): container finished" podID="d1301585-df11-436f-98dd-1350c1beb43f" containerID="a511c4dfb382f979cee06302bcbb34d4e12850173f191d5b53e4cfab03be9322" exitCode=60 Oct 29 22:30:40.373713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.373635 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerDied","Data":"a511c4dfb382f979cee06302bcbb34d4e12850173f191d5b53e4cfab03be9322"} Oct 29 22:30:40.374284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.374235 4543 scope.go:122] "RemoveContainer" containerID="a511c4dfb382f979cee06302bcbb34d4e12850173f191d5b53e4cfab03be9322" Oct 29 22:30:40.374797 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:40.374773 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz_odh-ai-gateway-infra(d1301585-df11-436f-98dd-1350c1beb43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" podUID="d1301585-df11-436f-98dd-1350c1beb43f" Oct 29 22:30:40.377533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.377514 4543 generic.go:356] "Generic (PLEG): container finished" podID="f03055aa-841f-4232-8009-046f61222149" containerID="4e5585f8141a6e1e692994666d8116a8550659dae4470ccd60c668008a8f58ea" exitCode=60 Oct 29 22:30:40.418252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.418220 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" Oct 29 22:30:40.462312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.462275 4543 scope.go:122] "RemoveContainer" containerID="a59defb975c15137f6bf1e2a700077bb749afd528b18e9c654cca4f786150c96" Oct 29 22:30:40.539698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.539617 4543 generic.go:356] "Generic (PLEG): container finished" podID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerID="149795c5e984be21019434a57224e52e0631b0745277d675a173f6b67841d8a2" exitCode=60 Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.569257 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" path="/var/lib/kubelet/pods/45274a01-b73e-490b-9b0c-1529c86e0f17/volumes" Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.570251 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerDied","Data":"4e5585f8141a6e1e692994666d8116a8550659dae4470ccd60c668008a8f58ea"} Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.570284 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l" event={"ID":"55e0645d-eeb1-44d2-a6fb-60c570f2dadc","Type":"ContainerDied","Data":"03ec6a280409c4d837a4576fe9a8e60aa7a14e772b24f873a33e58fac561ea0a"} Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.570296 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerDied","Data":"149795c5e984be21019434a57224e52e0631b0745277d675a173f6b67841d8a2"} Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.570737 4543 scope.go:122] "RemoveContainer" containerID="149795c5e984be21019434a57224e52e0631b0745277d675a173f6b67841d8a2" Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.574723 4543 scope.go:122] "RemoveContainer" containerID="342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20" Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.575932 4543 scope.go:122] "RemoveContainer" containerID="4e5585f8141a6e1e692994666d8116a8550659dae4470ccd60c668008a8f58ea" Oct 29 22:30:40.588450 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:40.576292 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z_odh-ai-gateway-infra(f03055aa-841f-4232-8009-046f61222149)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" podUID="f03055aa-841f-4232-8009-046f61222149" Oct 29 22:30:40.603151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.593950 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:40.648609 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:40.642188 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9_odh-ai-gateway-infra(a5b9973f-61e1-4a36-a5b0-9efd5e623d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" Oct 29 22:30:40.669485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.657994 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:40.685645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.680064 4543 scope.go:122] "RemoveContainer" containerID="e365eefa777c812950c5b098b8e2ef92600a97cd54bdc1a76ec9c72ab1994cb7" Oct 29 22:30:40.692930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.692505 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:40.703514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.703265 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:40.765192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.765166 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:40.787690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.787659 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:40.794334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.794308 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888550-2gn9l"] Oct 29 22:30:40.812886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.812866 4543 scope.go:122] "RemoveContainer" containerID="bac14d8d1051c904d8afe2a4ac5d8e86236c2120d3fb42ba3cfa53f008a6e7fd" Oct 29 22:30:40.841095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.840244 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:40.858848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.858821 4543 scope.go:122] "RemoveContainer" containerID="291dcd428bd61cb279454766d6a30cfc4f07d17522aee642446faa08fd0e91e4" Oct 29 22:30:40.870913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.870881 4543 scope.go:122] "RemoveContainer" containerID="fbd4adb4b55fe370deced90846e774d79c0016d27611193c97f706b53261d951" Oct 29 22:30:40.981999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.981960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\") pod \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " Oct 29 22:30:40.982194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.982032 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\" (UniqueName: \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\") pod \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\" (UID: \"93154a42-5f4b-40e7-bcb6-ed2c4d652144\") " Oct 29 22:30:40.982945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.982918 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca" pod "93154a42-5f4b-40e7-bcb6-ed2c4d652144" (UID: "93154a42-5f4b-40e7-bcb6-ed2c4d652144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:40.985960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:40.985934 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz" pod "93154a42-5f4b-40e7-bcb6-ed2c4d652144" (UID: "93154a42-5f4b-40e7-bcb6-ed2c4d652144"). InnerVolumeSpecName "kube-api-access-wrthz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:41.084594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.084564 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93154a42-5f4b-40e7-bcb6-ed2c4d652144-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:41.084594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.084592 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrthz\" (UniqueName: \"kubernetes.io/projected/93154a42-5f4b-40e7-bcb6-ed2c4d652144-kube-api-access-wrthz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:41.432124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.428168 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:41.564713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.564682 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" Oct 29 22:30:41.564887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.564685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m" event={"ID":"93154a42-5f4b-40e7-bcb6-ed2c4d652144","Type":"ContainerDied","Data":"66c39e18dd3571f459613246dc99b7eb2d64e3d9d4dded1f52200d5f681bc390"} Oct 29 22:30:41.564887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.564774 4543 scope.go:122] "RemoveContainer" containerID="b0668999607c6692e0259b0e3185acc337444d04bc6a13940bb142618c7262ea" Oct 29 22:30:41.570758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.570729 4543 generic.go:356] "Generic (PLEG): container finished" podID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerID="b2deb09ad7342528a9a75c272d0cba4cd3e69f50274fb0b15fe66f740779b4ad" exitCode=60 Oct 29 22:30:41.570869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.570809 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerDied","Data":"b2deb09ad7342528a9a75c272d0cba4cd3e69f50274fb0b15fe66f740779b4ad"} Oct 29 22:30:41.573383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.573355 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerStarted","Data":"0d473240708f97c553c2d23a3eac9eef8124eed917855691cf4a52bf09489160"} Oct 29 22:30:41.585508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.585454 4543 scope.go:122] "RemoveContainer" containerID="60ae6a52703ed7505bc28521e7b4c7101c27d2b9df397c9e862720cd43efab26" Oct 29 22:30:41.875922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.875888 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:41.885223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.885195 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888550-8cm7m"] Oct 29 22:30:41.965097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:41.961197 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:42.067213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.067181 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\" (UniqueName: \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\") pod \"f03055aa-841f-4232-8009-046f61222149\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " Oct 29 22:30:42.067607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.067577 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\") pod \"f03055aa-841f-4232-8009-046f61222149\" (UID: \"f03055aa-841f-4232-8009-046f61222149\") " Oct 29 22:30:42.075607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.074890 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd" pod "f03055aa-841f-4232-8009-046f61222149" (UID: "f03055aa-841f-4232-8009-046f61222149"). InnerVolumeSpecName "kube-api-access-84rfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.076625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.076583 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca" pod "f03055aa-841f-4232-8009-046f61222149" (UID: "f03055aa-841f-4232-8009-046f61222149"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.167837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.167773 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:42.170358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.170313 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rfd\" (UniqueName: \"kubernetes.io/projected/f03055aa-841f-4232-8009-046f61222149-kube-api-access-84rfd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.170358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.170338 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03055aa-841f-4232-8009-046f61222149-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.173896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.173874 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:42.179480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.179455 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:42.184937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.184912 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:42.190176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.190156 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:42.312440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.311013 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\" (UniqueName: \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\") pod \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " Oct 29 22:30:42.312440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.311170 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\" (UniqueName: \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\") pod \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " Oct 29 22:30:42.312440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.311306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\") pod \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\" (UID: \"01ffaa9e-56c1-4c3d-8441-4e38be2f089f\") " Oct 29 22:30:42.312440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.311502 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\") pod \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\" (UID: \"3a126320-b649-4e72-bd62-8a6a4f4f9b2f\") " Oct 29 22:30:42.312440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.311989 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca" pod "01ffaa9e-56c1-4c3d-8441-4e38be2f089f" (UID: "01ffaa9e-56c1-4c3d-8441-4e38be2f089f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.313230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.313172 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.313348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.313317 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca" pod "3a126320-b649-4e72-bd62-8a6a4f4f9b2f" (UID: "3a126320-b649-4e72-bd62-8a6a4f4f9b2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.319243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.319219 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c" pod "01ffaa9e-56c1-4c3d-8441-4e38be2f089f" (UID: "01ffaa9e-56c1-4c3d-8441-4e38be2f089f"). InnerVolumeSpecName "kube-api-access-vwg5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.335038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.334976 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b" pod "3a126320-b649-4e72-bd62-8a6a4f4f9b2f" (UID: "3a126320-b649-4e72-bd62-8a6a4f4f9b2f"). InnerVolumeSpecName "kube-api-access-8w76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.400117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.400059 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" path="/var/lib/kubelet/pods/55e0645d-eeb1-44d2-a6fb-60c570f2dadc/volumes" Oct 29 22:30:42.400904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.400882 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" path="/var/lib/kubelet/pods/93154a42-5f4b-40e7-bcb6-ed2c4d652144/volumes" Oct 29 22:30:42.413762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.413732 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\") pod \"d1301585-df11-436f-98dd-1350c1beb43f\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " Oct 29 22:30:42.413902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.413806 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\" (UniqueName: \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\") pod \"fe78a37f-554f-437b-a04d-ab5a9d250a12\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " Oct 29 22:30:42.414018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.413999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\" (UniqueName: \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\") pod \"d1301585-df11-436f-98dd-1350c1beb43f\" (UID: \"d1301585-df11-436f-98dd-1350c1beb43f\") " Oct 29 22:30:42.414134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.414048 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\" (UniqueName: \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\") pod \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " Oct 29 22:30:42.414134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.414111 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\") pod \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\" (UID: \"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b\") " Oct 29 22:30:42.414285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.414267 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\") pod \"fe78a37f-554f-437b-a04d-ab5a9d250a12\" (UID: \"fe78a37f-554f-437b-a04d-ab5a9d250a12\") " Oct 29 22:30:42.415224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.415197 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwg5c\" (UniqueName: \"kubernetes.io/projected/01ffaa9e-56c1-4c3d-8441-4e38be2f089f-kube-api-access-vwg5c\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.415343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.415228 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.415343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.415243 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w76b\" (UniqueName: \"kubernetes.io/projected/3a126320-b649-4e72-bd62-8a6a4f4f9b2f-kube-api-access-8w76b\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.415588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.415560 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca" pod "fe78a37f-554f-437b-a04d-ab5a9d250a12" (UID: "fe78a37f-554f-437b-a04d-ab5a9d250a12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.416109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.416055 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca" pod "d1301585-df11-436f-98dd-1350c1beb43f" (UID: "d1301585-df11-436f-98dd-1350c1beb43f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.416531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.416494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca" pod "a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" (UID: "a5b9973f-61e1-4a36-a5b0-9efd5e623d9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.421861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.421807 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m" pod "fe78a37f-554f-437b-a04d-ab5a9d250a12" (UID: "fe78a37f-554f-437b-a04d-ab5a9d250a12"). InnerVolumeSpecName "kube-api-access-5z28m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.425905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.425868 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz" pod "d1301585-df11-436f-98dd-1350c1beb43f" (UID: "d1301585-df11-436f-98dd-1350c1beb43f"). InnerVolumeSpecName "kube-api-access-l4mlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.428106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.428067 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb" pod "a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" (UID: "a5b9973f-61e1-4a36-a5b0-9efd5e623d9b"). InnerVolumeSpecName "kube-api-access-kwjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.517030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517002 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1301585-df11-436f-98dd-1350c1beb43f-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.517030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517030 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z28m\" (UniqueName: \"kubernetes.io/projected/fe78a37f-554f-437b-a04d-ab5a9d250a12-kube-api-access-5z28m\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.517445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517040 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mlz\" (UniqueName: \"kubernetes.io/projected/d1301585-df11-436f-98dd-1350c1beb43f-kube-api-access-l4mlz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.517445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517049 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwjjb\" (UniqueName: \"kubernetes.io/projected/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-kube-api-access-kwjjb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.517445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517059 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.517445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.517068 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe78a37f-554f-437b-a04d-ab5a9d250a12-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.604464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.604429 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" event={"ID":"3a126320-b649-4e72-bd62-8a6a4f4f9b2f","Type":"ContainerDied","Data":"6fe5d73f630b2408242d8d43e55ec8b2205cef022f2677397e76686e2f587d39"} Oct 29 22:30:42.604637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.604501 4543 scope.go:122] "RemoveContainer" containerID="b2deb09ad7342528a9a75c272d0cba4cd3e69f50274fb0b15fe66f740779b4ad" Oct 29 22:30:42.604637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.604448 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l" Oct 29 22:30:42.606894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.606864 4543 generic.go:356] "Generic (PLEG): container finished" podID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerID="0d473240708f97c553c2d23a3eac9eef8124eed917855691cf4a52bf09489160" exitCode=60 Oct 29 22:30:42.607006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.606959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerDied","Data":"0d473240708f97c553c2d23a3eac9eef8124eed917855691cf4a52bf09489160"} Oct 29 22:30:42.607419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.607403 4543 scope.go:122] "RemoveContainer" containerID="0d473240708f97c553c2d23a3eac9eef8124eed917855691cf4a52bf09489160" Oct 29 22:30:42.607807 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:42.607784 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw_odh-ai-gateway-infra(ef7cfdce-886f-47c2-a060-229f7b51c338)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" Oct 29 22:30:42.608816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.608793 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" event={"ID":"01ffaa9e-56c1-4c3d-8441-4e38be2f089f","Type":"ContainerDied","Data":"3cb59c33fc079523a43449c71b99590df8e7521100e1c74ab50c820a6cae3fc1"} Oct 29 22:30:42.608941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.608833 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn" Oct 29 22:30:42.610418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.610398 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" Oct 29 22:30:42.610516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.610403 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc" event={"ID":"fe78a37f-554f-437b-a04d-ab5a9d250a12","Type":"ContainerDied","Data":"0877b9cf1691a14c9d463edbf119fcedf136265001baefc216ba4faf2105b31f"} Oct 29 22:30:42.612266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.612247 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" Oct 29 22:30:42.612409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.612340 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz" event={"ID":"d1301585-df11-436f-98dd-1350c1beb43f","Type":"ContainerDied","Data":"e38c1f75a8707eaef1170e0f73ae84d0c47030992c6f5f08dc286fee1cddebc2"} Oct 29 22:30:42.614142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.614064 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" Oct 29 22:30:42.614256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.614098 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z" event={"ID":"f03055aa-841f-4232-8009-046f61222149","Type":"ContainerDied","Data":"b1ecabac1409f060c46fccddaa6cb19438e63d0dd1563a65271683d288499f78"} Oct 29 22:30:42.617652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.617635 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" Oct 29 22:30:42.617753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.617685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9" event={"ID":"a5b9973f-61e1-4a36-a5b0-9efd5e623d9b","Type":"ContainerDied","Data":"dfad44f1f3f6e19c9f7d8b2fd29873ed20a168a24800463f00d9bb5f7b924c77"} Oct 29 22:30:42.620837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.620813 4543 scope.go:122] "RemoveContainer" containerID="342a4b2d3147abbe879a0f3fa23cb7242dacfb79a609a736013c11d5a2b0ec20" Oct 29 22:30:42.636866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.636842 4543 scope.go:122] "RemoveContainer" containerID="ccc4efa290065f8983614920f71610ceee38541a97792765875c34e1d5113c38" Oct 29 22:30:42.648190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.648111 4543 scope.go:122] "RemoveContainer" containerID="ef4e8308013025b5e9bd7fd55eb70920200ddcce9b4ede8c4527d4064c4f4e6f" Oct 29 22:30:42.658992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.658970 4543 scope.go:122] "RemoveContainer" containerID="a511c4dfb382f979cee06302bcbb34d4e12850173f191d5b53e4cfab03be9322" Oct 29 22:30:42.669316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.669286 4543 scope.go:122] "RemoveContainer" containerID="4e5585f8141a6e1e692994666d8116a8550659dae4470ccd60c668008a8f58ea" Oct 29 22:30:42.680634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.680610 4543 scope.go:122] "RemoveContainer" containerID="149795c5e984be21019434a57224e52e0631b0745277d675a173f6b67841d8a2" Oct 29 22:30:42.720257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.720230 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:42.725850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.725825 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9ed94-29888550-ds6xn"] Oct 29 22:30:42.762028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.761925 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:42.770002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.769983 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-798a8f-29888550-5lh8z"] Oct 29 22:30:42.813704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.813682 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:42.818789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.818767 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888550-kmd2l"] Oct 29 22:30:42.909518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.909491 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:42.916139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.916119 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888550-n2lhc"] Oct 29 22:30:42.947909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.947855 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:42.955349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.955329 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888550-pdjrz"] Oct 29 22:30:42.972617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.972595 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:42.994746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:42.994711 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:43.004275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.004256 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-15154b-29888550-wtfk9"] Oct 29 22:30:43.681479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.681444 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:30:43.681943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.681500 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:30:43.842942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.842908 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:43.939998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.939908 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\" (UniqueName: \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\") pod \"ef7cfdce-886f-47c2-a060-229f7b51c338\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " Oct 29 22:30:43.940258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.940054 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\") pod \"ef7cfdce-886f-47c2-a060-229f7b51c338\" (UID: \"ef7cfdce-886f-47c2-a060-229f7b51c338\") " Oct 29 22:30:43.941132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.941047 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca" pod "ef7cfdce-886f-47c2-a060-229f7b51c338" (UID: "ef7cfdce-886f-47c2-a060-229f7b51c338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:43.943253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:43.943227 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww" pod "ef7cfdce-886f-47c2-a060-229f7b51c338" (UID: "ef7cfdce-886f-47c2-a060-229f7b51c338"). InnerVolumeSpecName "kube-api-access-fkwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:44.042329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.042289 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwww\" (UniqueName: \"kubernetes.io/projected/ef7cfdce-886f-47c2-a060-229f7b51c338-kube-api-access-fkwww\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.042329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.042323 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7cfdce-886f-47c2-a060-229f7b51c338-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.392723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.392684 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" path="/var/lib/kubelet/pods/01ffaa9e-56c1-4c3d-8441-4e38be2f089f/volumes" Oct 29 22:30:44.393487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.393467 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" path="/var/lib/kubelet/pods/3a126320-b649-4e72-bd62-8a6a4f4f9b2f/volumes" Oct 29 22:30:44.394105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.394089 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" path="/var/lib/kubelet/pods/a5b9973f-61e1-4a36-a5b0-9efd5e623d9b/volumes" Oct 29 22:30:44.395111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.395057 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1301585-df11-436f-98dd-1350c1beb43f" path="/var/lib/kubelet/pods/d1301585-df11-436f-98dd-1350c1beb43f/volumes" Oct 29 22:30:44.395653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.395637 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03055aa-841f-4232-8009-046f61222149" path="/var/lib/kubelet/pods/f03055aa-841f-4232-8009-046f61222149/volumes" Oct 29 22:30:44.396238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.396222 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" path="/var/lib/kubelet/pods/fe78a37f-554f-437b-a04d-ab5a9d250a12/volumes" Oct 29 22:30:44.649469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.649389 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" Oct 29 22:30:44.649469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.649406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw" event={"ID":"ef7cfdce-886f-47c2-a060-229f7b51c338","Type":"ContainerDied","Data":"6adf933959e5affb978ac8cd812775ffe58f55eb031fec5556b5bab6d0873869"} Oct 29 22:30:44.649469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.649468 4543 scope.go:122] "RemoveContainer" containerID="0d473240708f97c553c2d23a3eac9eef8124eed917855691cf4a52bf09489160" Oct 29 22:30:44.712687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.712658 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:44.724677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:44.724654 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-99e420-29888550-l2ksw"] Oct 29 22:30:45.230226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.230192 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:30:45.280948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.280906 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281201 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281271 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281452 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281507 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281541 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.281757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.281624 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\") pod \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\" (UID: \"5c08613e-80d1-423f-a3d3-4a5532c6dc54\") " Oct 29 22:30:45.283572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.283545 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.283768 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.283741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.283881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.283853 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:45.284067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.284036 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.284304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.284273 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.286851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.285503 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.286851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.285530 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.286851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.285542 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.286851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.285555 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.286851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.285568 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.289554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.289526 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "kube-api-access-6tsj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:45.290738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.290716 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.290890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.290734 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:45.292682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.292661 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo" pod "5c08613e-80d1-423f-a3d3-4a5532c6dc54" (UID: "5c08613e-80d1-423f-a3d3-4a5532c6dc54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:45.387270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.387232 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.387270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.387267 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tsj4\" (UniqueName: \"kubernetes.io/projected/5c08613e-80d1-423f-a3d3-4a5532c6dc54-kube-api-access-6tsj4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.387514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.387282 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.387514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.387297 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c08613e-80d1-423f-a3d3-4a5532c6dc54-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.664651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.664614 4543 generic.go:356] "Generic (PLEG): container finished" podID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerID="439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4" exitCode=0 Oct 29 22:30:45.664836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.664692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerDied","Data":"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4"} Oct 29 22:30:45.664836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.664708 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" Oct 29 22:30:45.664836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.664755 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd" event={"ID":"5c08613e-80d1-423f-a3d3-4a5532c6dc54","Type":"ContainerDied","Data":"aacb1bf7c132a1515bc4c10ff1914725e84b99a2af56a717fd98f9adfc467abf"} Oct 29 22:30:45.664836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.664777 4543 scope.go:122] "RemoveContainer" containerID="439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4" Oct 29 22:30:45.681086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.681055 4543 scope.go:122] "RemoveContainer" containerID="6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.705765 4543 scope.go:122] "RemoveContainer" containerID="439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:45.706227 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4\": container with ID starting with 439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4 not found: ID does not exist" containerID="439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.706277 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4"} err="failed to get container status \"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4\": rpc error: code = NotFound desc = could not find container \"439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4\": container with ID starting with 439d5286f2f19469a07c72690964ccf5edf161b0eed8a70b6e679c394dc4d5d4 not found: ID does not exist" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.706301 4543 scope.go:122] "RemoveContainer" containerID="6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:45.706671 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932\": container with ID starting with 6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932 not found: ID does not exist" containerID="6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932" Oct 29 22:30:45.717065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.706708 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932"} err="failed to get container status \"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932\": rpc error: code = NotFound desc = could not find container \"6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932\": container with ID starting with 6489932a0aaf2e755817d8ea547d4b01684719a3a928f57082127f06d9bb7932 not found: ID does not exist" Oct 29 22:30:45.841097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:45.806844 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:30:46.009656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:46.009144 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-zc8cd"] Oct 29 22:30:46.391853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:46.391779 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" path="/var/lib/kubelet/pods/5c08613e-80d1-423f-a3d3-4a5532c6dc54/volumes" Oct 29 22:30:46.392550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:46.392528 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" path="/var/lib/kubelet/pods/ef7cfdce-886f-47c2-a060-229f7b51c338/volumes" Oct 29 22:30:46.999197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:46.999065 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": read tcp 10.133.0.2:53362->10.133.1.114:15021: read: connection reset by peer" start-of-body= Oct 29 22:30:46.999197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:46.999177 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": read tcp 10.133.0.2:53362->10.133.1.114:15021: read: connection reset by peer" Oct 29 22:30:47.688980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.688945 4543 generic.go:356] "Generic (PLEG): container finished" podID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerID="177453861d86c20095c57375c07acd5cc3e6d96dbb79749d98188d5b6b241009" exitCode=0 Oct 29 22:30:47.689179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.689022 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerDied","Data":"177453861d86c20095c57375c07acd5cc3e6d96dbb79749d98188d5b6b241009"} Oct 29 22:30:47.689522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.689500 4543 scope.go:122] "RemoveContainer" containerID="177453861d86c20095c57375c07acd5cc3e6d96dbb79749d98188d5b6b241009" Oct 29 22:30:47.691633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.691601 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:47.691732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.691676 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:47.691732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:47.691708 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:30:48.700925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:48.700889 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerStarted","Data":"9448e0f1c921c054ec3b61c5aee3ee594a78969379654cac4f895ad5f4e5af63"} Oct 29 22:30:49.546102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:49.541372 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:30:49.553226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:49.553198 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body= Oct 29 22:30:49.553387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:49.553321 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" Oct 29 22:30:50.551102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:50.550415 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body= Oct 29 22:30:50.551102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:50.550471 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" Oct 29 22:30:51.553109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:51.551282 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body= Oct 29 22:30:51.553109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:51.551395 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" Oct 29 22:30:52.549101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:52.547145 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body= Oct 29 22:30:52.549101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:52.547243 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" Oct 29 22:30:53.553523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:53.553451 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" start-of-body= Oct 29 22:30:53.554026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:53.553527 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.114:15021/healthz/ready\": dial tcp 10.133.1.114:15021: connect: connection refused" Oct 29 22:30:54.542006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:54.541978 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:30:54.542359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:54.542338 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:30:54.543144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:30:54.543124 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:30:57.969370 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:30:57.969328 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:31:05.762397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.762317 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv"] Oct 29 22:31:05.763436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763415 4543 cpu_manager.go:414] "containerMap: removing container" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763437 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763447 4543 cpu_manager.go:414] "containerMap: removing container" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763452 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763465 4543 cpu_manager.go:414] "containerMap: removing container" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763470 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763480 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763484 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.763497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763496 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763500 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763507 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763512 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763522 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763526 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763531 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763536 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763545 4543 cpu_manager.go:414] "containerMap: removing container" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763549 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763564 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763569 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763578 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763582 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763592 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763597 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763609 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763614 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763620 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763624 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763629 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763633 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763642 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763646 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763654 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763660 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763672 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763678 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763687 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763694 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763707 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763714 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763727 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763732 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763742 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763746 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763756 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763761 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.763752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763768 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763772 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763784 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763788 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763798 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763802 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763971 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763980 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763988 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763993 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.763999 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764006 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764012 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764017 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764024 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764029 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764037 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764042 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764047 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764052 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764059 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764065 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764086 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c08613e-80d1-423f-a3d3-4a5532c6dc54" containerName="istio-proxy" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764092 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03055aa-841f-4232-8009-046f61222149" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764100 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764106 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764113 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764118 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764123 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764129 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764134 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764139 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a126320-b649-4e72-bd62-8a6a4f4f9b2f" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764146 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b9973f-61e1-4a36-a5b0-9efd5e623d9b" containerName="cleanup" Oct 29 22:31:05.765019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.764153 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1301585-df11-436f-98dd-1350c1beb43f" containerName="cleanup" Oct 29 22:31:05.770386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.770362 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.773519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.773498 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv"] Oct 29 22:31:05.787616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.787598 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:05.800096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.800054 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:31:05.860265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.860236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hld\" (UniqueName: \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.860434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.860408 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.860683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.860668 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.961906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.961878 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.962115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.961942 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hld\" (UniqueName: \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.962115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.962009 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.962678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.962654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.964446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.964426 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:05.987989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:05.987967 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hld\" (UniqueName: \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\") pod \"maas-api-revoke-sub-8badc926-5jwlv\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:06.090812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:06.090726 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:06.427516 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:31:06.427490 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb97bda_7946_4104_93ea_db0a353bea3b.slice/crio-85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9 WatchSource:0}: Error finding container 85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9: Status 404 returned error can't find the container with id 85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9 Oct 29 22:31:06.447097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:06.428227 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv"] Oct 29 22:31:07.357152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:07.330452 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" event={"ID":"bcb97bda-7946-4104-93ea-db0a353bea3b","Type":"ContainerStarted","Data":"241bc851dd14bbed2eada9e71ef64b26ac68bbb3dec3f6ca30b1a3fb365fbaf2"} Oct 29 22:31:07.357152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:07.330522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" event={"ID":"bcb97bda-7946-4104-93ea-db0a353bea3b","Type":"ContainerStarted","Data":"85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9"} Oct 29 22:31:07.464101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:07.425040 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" podStartSLOduration=2.425020507 podStartE2EDuration="2.425020507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:07.423969699 +0000 UTC m=+2929.165900373" watchObservedRunningTime="2026-10-29 22:31:07.425020507 +0000 UTC m=+2929.166951194" Oct 29 22:31:08.364545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:08.364500 4543 generic.go:356] "Generic (PLEG): container finished" podID="bcb97bda-7946-4104-93ea-db0a353bea3b" containerID="241bc851dd14bbed2eada9e71ef64b26ac68bbb3dec3f6ca30b1a3fb365fbaf2" exitCode=0 Oct 29 22:31:08.365050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:08.364676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" event={"ID":"bcb97bda-7946-4104-93ea-db0a353bea3b","Type":"ContainerDied","Data":"241bc851dd14bbed2eada9e71ef64b26ac68bbb3dec3f6ca30b1a3fb365fbaf2"} Oct 29 22:31:09.633522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.633462 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:09.801558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.801526 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\") pod \"bcb97bda-7946-4104-93ea-db0a353bea3b\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " Oct 29 22:31:09.801733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.801702 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\" (UniqueName: \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\") pod \"bcb97bda-7946-4104-93ea-db0a353bea3b\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " Oct 29 22:31:09.801836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.801818 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\") pod \"bcb97bda-7946-4104-93ea-db0a353bea3b\" (UID: \"bcb97bda-7946-4104-93ea-db0a353bea3b\") " Oct 29 22:31:09.802427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.802026 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca" pod "bcb97bda-7946-4104-93ea-db0a353bea3b" (UID: "bcb97bda-7946-4104-93ea-db0a353bea3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:09.803472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.803444 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:09.804579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.804550 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca" pod "bcb97bda-7946-4104-93ea-db0a353bea3b" (UID: "bcb97bda-7946-4104-93ea-db0a353bea3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:09.804692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.804676 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld" pod "bcb97bda-7946-4104-93ea-db0a353bea3b" (UID: "bcb97bda-7946-4104-93ea-db0a353bea3b"). InnerVolumeSpecName "kube-api-access-87hld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:09.904507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.904450 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87hld\" (UniqueName: \"kubernetes.io/projected/bcb97bda-7946-4104-93ea-db0a353bea3b-kube-api-access-87hld\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:09.904507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:09.904502 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb97bda-7946-4104-93ea-db0a353bea3b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:10.390444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:10.390400 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" Oct 29 22:31:10.393748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:10.393713 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv" event={"ID":"bcb97bda-7946-4104-93ea-db0a353bea3b","Type":"ContainerDied","Data":"85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9"} Oct 29 22:31:10.393921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:10.393759 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85beb8dc6755927bb2f5d053794f67e1be7d4262cbdd7570827739173ed433a9" Oct 29 22:31:13.389262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:13.389224 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:31:13.389877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:13.389310 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:14.653926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.653892 4543 generic.go:356] "Generic (PLEG): container finished" podID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerID="a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5" exitCode=0 Oct 29 22:31:14.653926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.653920 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerDied","Data":"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5"} Oct 29 22:31:14.654506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.654320 4543 scope.go:122] "RemoveContainer" containerID="a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5" Oct 29 22:31:14.656543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.656514 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:14.656647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.656581 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:14.656647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:14.656631 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:31:15.666787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:15.666744 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerStarted","Data":"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29"} Oct 29 22:31:16.317732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.317697 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:31:16.317929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.317896 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body= Oct 29 22:31:16.317974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.317948 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" Oct 29 22:31:16.560415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.560382 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644"] Oct 29 22:31:16.562378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562345 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:16.562378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562377 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562407 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562416 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562478 4543 cpu_manager.go:414] "containerMap: removing container" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562486 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562501 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562510 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562528 4543 cpu_manager.go:414] "containerMap: removing container" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562535 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45274a01-b73e-490b-9b0c-1529c86e0f17" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562554 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bcb97bda-7946-4104-93ea-db0a353bea3b" containerName="revoke-keys" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562562 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb97bda-7946-4104-93ea-db0a353bea3b" containerName="revoke-keys" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562579 4543 cpu_manager.go:414] "containerMap: removing container" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:16.562605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562588 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffaa9e-56c1-4c3d-8441-4e38be2f089f" containerName="cleanup" Oct 29 22:31:16.563030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562838 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e0645d-eeb1-44d2-a6fb-60c570f2dadc" containerName="cleanup" Oct 29 22:31:16.563030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562852 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93154a42-5f4b-40e7-bcb6-ed2c4d652144" containerName="cleanup" Oct 29 22:31:16.563030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562866 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb97bda-7946-4104-93ea-db0a353bea3b" containerName="revoke-keys" Oct 29 22:31:16.563030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562889 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe78a37f-554f-437b-a04d-ab5a9d250a12" containerName="cleanup" Oct 29 22:31:16.563030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.562908 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7cfdce-886f-47c2-a060-229f7b51c338" containerName="cleanup" Oct 29 22:31:16.567018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.566994 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644"] Oct 29 22:31:16.567183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.567148 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.580254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.580225 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.580571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.580541 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.580697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.580634 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7nl\" (UniqueName: \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.596895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.596873 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:16.597047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.597026 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:31:16.682036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.682007 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7nl\" (UniqueName: \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.682428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.682215 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.682428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.682313 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.682981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.682962 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.684818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.684797 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.707457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.707435 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7nl\" (UniqueName: \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\") pod \"maas-api-revoke-sub-6f8b7824-78644\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:16.888867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:16.888775 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:17.259826 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:31:17.259796 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff603f0_e6d0_4568_b64d_f8e6428e4ff8.slice/crio-ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2 WatchSource:0}: Error finding container ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2: Status 404 returned error can't find the container with id ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2 Oct 29 22:31:17.264868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:17.264847 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644"] Oct 29 22:31:17.317321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:17.317298 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body= Oct 29 22:31:17.317414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:17.317344 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" Oct 29 22:31:17.690809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:17.690732 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" event={"ID":"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8","Type":"ContainerStarted","Data":"ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2"} Oct 29 22:31:18.318018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:18.317977 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body= Oct 29 22:31:18.318244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:18.318038 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" Oct 29 22:31:18.702397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:18.702364 4543 generic.go:356] "Generic (PLEG): container finished" podID="4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" containerID="2ca5b95f9ead51a2fb5cd23856bbb79b7fcf82af37bf40d4e167308e37a0fd39" exitCode=0 Oct 29 22:31:18.702773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:18.702405 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" event={"ID":"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8","Type":"ContainerDied","Data":"2ca5b95f9ead51a2fb5cd23856bbb79b7fcf82af37bf40d4e167308e37a0fd39"} Oct 29 22:31:19.317195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:19.317159 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body= Oct 29 22:31:19.317366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:19.317211 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" Oct 29 22:31:19.990912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:19.990850 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:20.075120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.075089 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\") pod \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " Oct 29 22:31:20.075381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.075363 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\") pod \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " Oct 29 22:31:20.075720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.075700 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\" (UniqueName: \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\") pod \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\" (UID: \"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8\") " Oct 29 22:31:20.075902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.075742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca" pod "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" (UID: "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:20.077178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.077153 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:20.077946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.077922 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca" pod "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" (UID: "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:20.078228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.078206 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl" pod "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" (UID: "4ff603f0-e6d0-4568-b64d-f8e6428e4ff8"). InnerVolumeSpecName "kube-api-access-2b7nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:20.179085 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.179038 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7nl\" (UniqueName: \"kubernetes.io/projected/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-kube-api-access-2b7nl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:20.179085 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.179067 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:20.317364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.317278 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" start-of-body= Oct 29 22:31:20.317364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.317341 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.116:15021/healthz/ready\": dial tcp 10.133.1.116:15021: connect: connection refused" Oct 29 22:31:20.722305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.722277 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" Oct 29 22:31:20.722475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.722317 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644" event={"ID":"4ff603f0-e6d0-4568-b64d-f8e6428e4ff8","Type":"ContainerDied","Data":"ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2"} Oct 29 22:31:20.722475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:20.722364 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ded8617137dc4aca9858081662ffe047b2c61936463a4cba0d4607dd21e6b9b2" Oct 29 22:31:21.322374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:21.322346 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:31:21.322869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:21.322847 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:31:21.323668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:21.323650 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:31:23.675191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:23.675137 4543 scope.go:122] "RemoveContainer" containerID="fb7e9dab1c7eec8ecc92b7bb225f82dc89496f9d8ca5a782a1a735019adb819e" Oct 29 22:31:23.690140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:23.690110 4543 scope.go:122] "RemoveContainer" containerID="06e731f8e3664246de6616c5fa187d7cc47cfad53badf7eb630c5c5056de8d11" Oct 29 22:31:27.962626 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:31:27.962591 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:31:29.277730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:29.277696 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:31:34.278245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.278183 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" podUID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" containerName="payload-processing" containerID="cri-o://a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c" gracePeriod=25 Oct 29 22:31:34.584497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.584470 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:31:34.679035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\") pod \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " Oct 29 22:31:34.679225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\") pod \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " Oct 29 22:31:34.679324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679310 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\") pod \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " Oct 29 22:31:34.679412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume" pod "f952f7b4-61c2-46c2-9d52-d40a0b889ea1" (UID: "f952f7b4-61c2-46c2-9d52-d40a0b889ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:34.679478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679422 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\") pod \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\" (UID: \"f952f7b4-61c2-46c2-9d52-d40a0b889ea1\") " Oct 29 22:31:34.679790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.679769 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume" pod "f952f7b4-61c2-46c2-9d52-d40a0b889ea1" (UID: "f952f7b4-61c2-46c2-9d52-d40a0b889ea1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:34.680133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.680115 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:34.680198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.680136 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:34.682215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.682193 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials" pod "f952f7b4-61c2-46c2-9d52-d40a0b889ea1" (UID: "f952f7b4-61c2-46c2-9d52-d40a0b889ea1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:34.682321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.682225 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq" pod "f952f7b4-61c2-46c2-9d52-d40a0b889ea1" (UID: "f952f7b4-61c2-46c2-9d52-d40a0b889ea1"). InnerVolumeSpecName "kube-api-access-8vtxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:34.781284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.781259 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vtxq\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-kube-api-access-8vtxq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:34.781284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:34.781283 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f952f7b4-61c2-46c2-9d52-d40a0b889ea1-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:35.008931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.008903 4543 generic.go:356] "Generic (PLEG): container finished" podID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" containerID="a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c" exitCode=0 Oct 29 22:31:35.009109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.008957 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" Oct 29 22:31:35.009109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.008948 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" event={"ID":"f952f7b4-61c2-46c2-9d52-d40a0b889ea1","Type":"ContainerDied","Data":"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c"} Oct 29 22:31:35.009109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.009025 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf" event={"ID":"f952f7b4-61c2-46c2-9d52-d40a0b889ea1","Type":"ContainerDied","Data":"46fe70859b1b8b448e6c59c7094ff5badaef6444c86c34003d485123fbd1d0f5"} Oct 29 22:31:35.009109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.009059 4543 scope.go:122] "RemoveContainer" containerID="a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c" Oct 29 22:31:35.024786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.024761 4543 scope.go:122] "RemoveContainer" containerID="a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c" Oct 29 22:31:35.025158 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:31:35.025133 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c\": container with ID starting with a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c not found: ID does not exist" containerID="a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c" Oct 29 22:31:35.025259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.025167 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c"} err="failed to get container status \"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c\": rpc error: code = NotFound desc = could not find container \"a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c\": container with ID starting with a967e2e27a49bfac736cfc353c072adb400204fd64808823b8f006de58c05a4c not found: ID does not exist" Oct 29 22:31:35.070784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.070763 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:31:35.073391 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:31:35.073372 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf952f7b4_61c2_46c2_9d52_d40a0b889ea1.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:31:35.084279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:35.084256 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0c3d9a/payload-processing-external-model-e2e-extmod-a-0c3d9a-65cdrlbwf"] Oct 29 22:31:36.390059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:36.390024 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" path="/var/lib/kubelet/pods/f952f7b4-61c2-46c2-9d52-d40a0b889ea1/volumes" Oct 29 22:31:43.050917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:43.050853 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v"] Oct 29 22:31:43.184089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:43.184035 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8af1539-78f6v"] Oct 29 22:31:44.389482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:44.389447 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a34efe0-9615-4fd4-940f-c70aff824c78" path="/var/lib/kubelet/pods/9a34efe0-9615-4fd4-940f-c70aff824c78/volumes" Oct 29 22:31:46.102559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.102524 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j"] Oct 29 22:31:46.103613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103592 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" containerName="revoke-keys" Oct 29 22:31:46.103613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103614 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" containerName="revoke-keys" Oct 29 22:31:46.103767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103663 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" containerName="payload-processing" Oct 29 22:31:46.103767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103668 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" containerName="payload-processing" Oct 29 22:31:46.103837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103800 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f952f7b4-61c2-46c2-9d52-d40a0b889ea1" containerName="payload-processing" Oct 29 22:31:46.103837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.103814 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" containerName="revoke-keys" Oct 29 22:31:46.109478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.109461 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.127602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.127579 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j"] Oct 29 22:31:46.134162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.134139 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:46.134286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.134244 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:31:46.189242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.189215 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.189407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.189382 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.189489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.189411 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplvk\" (UniqueName: \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.291577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.291547 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.291790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.291655 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.291790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.291673 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplvk\" (UniqueName: \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.292488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.292459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.294388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.294364 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.318971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.318943 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplvk\" (UniqueName: \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\") pod \"maas-api-revoke-sub-71fe183b-4ns8j\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.426749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.426687 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:46.798293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:46.798259 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j"] Oct 29 22:31:47.100504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.100440 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km"] Oct 29 22:31:47.107523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.107503 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.124028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.124005 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km"] Oct 29 22:31:47.221989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.221961 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.222167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.222028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.222218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.222174 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk22p\" (UniqueName: \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.294467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.294435 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" event={"ID":"84d91908-86e6-4da8-8c2b-3841df3ca4c7","Type":"ContainerStarted","Data":"1cd779d0c155221271e6b557f6835fb394acc1ebb5a34a8936754c5d502d17dd"} Oct 29 22:31:47.294617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.294490 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" event={"ID":"84d91908-86e6-4da8-8c2b-3841df3ca4c7","Type":"ContainerStarted","Data":"929c781fc31796d9baeb2ef84da2075f13cadab622660c758d4956206be6243b"} Oct 29 22:31:47.324117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.324046 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.324221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.324169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.324221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.324210 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk22p\" (UniqueName: \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.324799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.324776 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.326548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.326525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.343319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.343278 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" podStartSLOduration=1.343265654 podStartE2EDuration="1.343265654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:47.339436461 +0000 UTC m=+2969.081367135" watchObservedRunningTime="2026-10-29 22:31:47.343265654 +0000 UTC m=+2969.085196326" Oct 29 22:31:47.350336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.350317 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk22p\" (UniqueName: \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\") pod \"maas-api-revoke-sub-1b6752a6-zm2km\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.424987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.424939 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:47.747359 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:31:47.747327 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf9ceb_e7a4_4ac5_bf8a_ea9c088d7792.slice/crio-4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200 WatchSource:0}: Error finding container 4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200: Status 404 returned error can't find the container with id 4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200 Oct 29 22:31:47.747990 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:47.747969 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km"] Oct 29 22:31:48.308116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:48.308029 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" event={"ID":"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792","Type":"ContainerStarted","Data":"ab127017f5d4ab0d79c4f8b91f50f6c8a81cd1d68d8d32e7223f0dd3e339e32c"} Oct 29 22:31:48.308481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:48.308118 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" event={"ID":"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792","Type":"ContainerStarted","Data":"4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200"} Oct 29 22:31:48.309906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:48.309865 4543 generic.go:356] "Generic (PLEG): container finished" podID="84d91908-86e6-4da8-8c2b-3841df3ca4c7" containerID="1cd779d0c155221271e6b557f6835fb394acc1ebb5a34a8936754c5d502d17dd" exitCode=0 Oct 29 22:31:48.310021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:48.309940 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" event={"ID":"84d91908-86e6-4da8-8c2b-3841df3ca4c7","Type":"ContainerDied","Data":"1cd779d0c155221271e6b557f6835fb394acc1ebb5a34a8936754c5d502d17dd"} Oct 29 22:31:48.358349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:48.358284 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" podStartSLOduration=1.358272804 podStartE2EDuration="1.358272804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:48.354494896 +0000 UTC m=+2970.096425568" watchObservedRunningTime="2026-10-29 22:31:48.358272804 +0000 UTC m=+2970.100203477" Oct 29 22:31:49.321461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.321425 4543 generic.go:356] "Generic (PLEG): container finished" podID="00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" containerID="ab127017f5d4ab0d79c4f8b91f50f6c8a81cd1d68d8d32e7223f0dd3e339e32c" exitCode=0 Oct 29 22:31:49.321864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.321505 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" event={"ID":"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792","Type":"ContainerDied","Data":"ab127017f5d4ab0d79c4f8b91f50f6c8a81cd1d68d8d32e7223f0dd3e339e32c"} Oct 29 22:31:49.514496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.514472 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:49.581650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.581581 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\" (UniqueName: \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\") pod \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " Oct 29 22:31:49.581807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.581657 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\") pod \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " Oct 29 22:31:49.581807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.581709 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\") pod \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\" (UID: \"84d91908-86e6-4da8-8c2b-3841df3ca4c7\") " Oct 29 22:31:49.582135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.582100 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca" pod "84d91908-86e6-4da8-8c2b-3841df3ca4c7" (UID: "84d91908-86e6-4da8-8c2b-3841df3ca4c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:49.582594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.582573 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:49.584518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.584493 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca" pod "84d91908-86e6-4da8-8c2b-3841df3ca4c7" (UID: "84d91908-86e6-4da8-8c2b-3841df3ca4c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:49.584518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.584507 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk" pod "84d91908-86e6-4da8-8c2b-3841df3ca4c7" (UID: "84d91908-86e6-4da8-8c2b-3841df3ca4c7"). InnerVolumeSpecName "kube-api-access-zplvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:49.684037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.684004 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplvk\" (UniqueName: \"kubernetes.io/projected/84d91908-86e6-4da8-8c2b-3841df3ca4c7-kube-api-access-zplvk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:49.684037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:49.684037 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d91908-86e6-4da8-8c2b-3841df3ca4c7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:50.332406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.332368 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" event={"ID":"84d91908-86e6-4da8-8c2b-3841df3ca4c7","Type":"ContainerDied","Data":"929c781fc31796d9baeb2ef84da2075f13cadab622660c758d4956206be6243b"} Oct 29 22:31:50.332838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.332428 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929c781fc31796d9baeb2ef84da2075f13cadab622660c758d4956206be6243b" Oct 29 22:31:50.332838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.332390 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j" Oct 29 22:31:50.527266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.527240 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:50.624176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.622827 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\" (UniqueName: \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\") pod \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " Oct 29 22:31:50.624176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.622899 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\") pod \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " Oct 29 22:31:50.624176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.623064 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\") pod \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\" (UID: \"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792\") " Oct 29 22:31:50.638566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.638530 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca" pod "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" (UID: "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:50.847257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.835065 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:50.847257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.847197 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca" pod "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" (UID: "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:50.967185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.957008 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:50.995975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:50.995904 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p" pod "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" (UID: "00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792"). InnerVolumeSpecName "kube-api-access-rk22p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:51.059179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:51.059144 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk22p\" (UniqueName: \"kubernetes.io/projected/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792-kube-api-access-rk22p\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:51.344013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:51.343928 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" Oct 29 22:31:51.344013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:51.343956 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km" event={"ID":"00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792","Type":"ContainerDied","Data":"4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200"} Oct 29 22:31:51.344013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:51.344005 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4075571fabd45f65e125c239dcb878a7f02b632d3672cfbd349fb96998477200" Oct 29 22:31:56.040354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.040325 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv"] Oct 29 22:31:56.052875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.052850 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20815ee-7n9zv"] Oct 29 22:31:56.389027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.388955 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91586ca4-47ae-4f76-938c-4e4d39f99a83" path="/var/lib/kubelet/pods/91586ca4-47ae-4f76-938c-4e4d39f99a83/volumes" Oct 29 22:31:56.817447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.817420 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq"] Oct 29 22:31:56.818400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818384 4543 cpu_manager.go:414] "containerMap: removing container" podUID="00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" containerName="revoke-keys" Oct 29 22:31:56.818453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818402 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" containerName="revoke-keys" Oct 29 22:31:56.818453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818420 4543 cpu_manager.go:414] "containerMap: removing container" podUID="84d91908-86e6-4da8-8c2b-3841df3ca4c7" containerName="revoke-keys" Oct 29 22:31:56.818453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818425 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d91908-86e6-4da8-8c2b-3841df3ca4c7" containerName="revoke-keys" Oct 29 22:31:56.818604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818594 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" containerName="revoke-keys" Oct 29 22:31:56.818640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.818608 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d91908-86e6-4da8-8c2b-3841df3ca4c7" containerName="revoke-keys" Oct 29 22:31:56.822831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.822811 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:56.843351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.843324 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq"] Oct 29 22:31:56.847907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.847882 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:31:56.848039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.848003 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:31:56.967495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.967468 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:56.967697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.967675 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:56.967852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:56.967710 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5g6\" (UniqueName: \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.073723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.071649 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.073723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.071731 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5g6\" (UniqueName: \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.073723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.072019 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.074476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.073910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.078361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.076137 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.106292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.106266 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5g6\" (UniqueName: \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\") pod \"maas-api-revoke-sub-90416f56-4zqfq\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.139963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.139934 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:57.455615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.455587 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq"] Oct 29 22:31:57.456260 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:31:57.456227 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2b9bf2_b313_4f1c_8e40_570049284e49.slice/crio-714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0 WatchSource:0}: Error finding container 714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0: Status 404 returned error can't find the container with id 714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0 Oct 29 22:31:57.796733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.796708 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc"] Oct 29 22:31:57.803481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.803460 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.821395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.821373 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc"] Oct 29 22:31:57.884952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.884917 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.885122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.885053 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk7n\" (UniqueName: \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.885309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.885284 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.962132 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:31:57.962056 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:31:57.986983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.986960 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.987104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.987037 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk7n\" (UniqueName: \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.987161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.987152 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.987782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.987761 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:57.989438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:57.989418 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:58.012451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.012402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk7n\" (UniqueName: \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\") pod \"maas-api-revoke-sub-6a790551-2rttc\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:58.123918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.123897 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:31:58.441469 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:31:58.441438 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a325ac_8170_44a7_a768_a4ca86e5890d.slice/crio-c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3 WatchSource:0}: Error finding container c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3: Status 404 returned error can't find the container with id c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3 Oct 29 22:31:58.445185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.445160 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc"] Oct 29 22:31:58.464978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.464952 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" event={"ID":"ed2b9bf2-b313-4f1c-8e40-570049284e49","Type":"ContainerStarted","Data":"7d40fa9593ede4347fa981b1b51e9dfa52fa52775cc0521f753e72a663705426"} Oct 29 22:31:58.465113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.465007 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" event={"ID":"ed2b9bf2-b313-4f1c-8e40-570049284e49","Type":"ContainerStarted","Data":"714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0"} Oct 29 22:31:58.466610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:58.466587 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" event={"ID":"59a325ac-8170-44a7-a768-a4ca86e5890d","Type":"ContainerStarted","Data":"c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3"} Oct 29 22:31:59.477611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.477583 4543 generic.go:356] "Generic (PLEG): container finished" podID="ed2b9bf2-b313-4f1c-8e40-570049284e49" containerID="7d40fa9593ede4347fa981b1b51e9dfa52fa52775cc0521f753e72a663705426" exitCode=0 Oct 29 22:31:59.477983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.477662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" event={"ID":"ed2b9bf2-b313-4f1c-8e40-570049284e49","Type":"ContainerDied","Data":"7d40fa9593ede4347fa981b1b51e9dfa52fa52775cc0521f753e72a663705426"} Oct 29 22:31:59.479662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.479637 4543 generic.go:356] "Generic (PLEG): container finished" podID="59a325ac-8170-44a7-a768-a4ca86e5890d" containerID="9b2752b2192f765370b684b5cfecc4eee929f93e8a43a6927c10bc2c8adbdd9d" exitCode=0 Oct 29 22:31:59.479786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.479684 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" event={"ID":"59a325ac-8170-44a7-a768-a4ca86e5890d","Type":"ContainerDied","Data":"9b2752b2192f765370b684b5cfecc4eee929f93e8a43a6927c10bc2c8adbdd9d"} Oct 29 22:31:59.659681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.659655 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:31:59.820109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.820017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\") pod \"ed2b9bf2-b313-4f1c-8e40-570049284e49\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " Oct 29 22:31:59.820109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.820092 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\" (UniqueName: \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\") pod \"ed2b9bf2-b313-4f1c-8e40-570049284e49\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " Oct 29 22:31:59.820342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.820151 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\") pod \"ed2b9bf2-b313-4f1c-8e40-570049284e49\" (UID: \"ed2b9bf2-b313-4f1c-8e40-570049284e49\") " Oct 29 22:31:59.820554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.820512 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca" pod "ed2b9bf2-b313-4f1c-8e40-570049284e49" (UID: "ed2b9bf2-b313-4f1c-8e40-570049284e49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:59.821135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.821108 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:59.822384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.822362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca" pod "ed2b9bf2-b313-4f1c-8e40-570049284e49" (UID: "ed2b9bf2-b313-4f1c-8e40-570049284e49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:31:59.822497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.822478 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6" pod "ed2b9bf2-b313-4f1c-8e40-570049284e49" (UID: "ed2b9bf2-b313-4f1c-8e40-570049284e49"). InnerVolumeSpecName "kube-api-access-xj5g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:59.922917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.922891 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2b9bf2-b313-4f1c-8e40-570049284e49-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:31:59.922917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:31:59.922916 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5g6\" (UniqueName: \"kubernetes.io/projected/ed2b9bf2-b313-4f1c-8e40-570049284e49-kube-api-access-xj5g6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:00.490873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.490838 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" Oct 29 22:32:00.490873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.490850 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq" event={"ID":"ed2b9bf2-b313-4f1c-8e40-570049284e49","Type":"ContainerDied","Data":"714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0"} Oct 29 22:32:00.491575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.490907 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714336ee8de111e852fbee00bcc9b47d5bffdd762e8c829a7124beca7b3bb5a0" Oct 29 22:32:00.687921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.687893 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:32:00.842442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.842370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\" (UniqueName: \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\") pod \"59a325ac-8170-44a7-a768-a4ca86e5890d\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " Oct 29 22:32:00.842585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.842560 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\") pod \"59a325ac-8170-44a7-a768-a4ca86e5890d\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " Oct 29 22:32:00.842665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.842650 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\") pod \"59a325ac-8170-44a7-a768-a4ca86e5890d\" (UID: \"59a325ac-8170-44a7-a768-a4ca86e5890d\") " Oct 29 22:32:00.842916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.842894 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca" pod "59a325ac-8170-44a7-a768-a4ca86e5890d" (UID: "59a325ac-8170-44a7-a768-a4ca86e5890d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:00.843298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.843279 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:00.844716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.844693 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n" pod "59a325ac-8170-44a7-a768-a4ca86e5890d" (UID: "59a325ac-8170-44a7-a768-a4ca86e5890d"). InnerVolumeSpecName "kube-api-access-6dk7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:00.844776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.844723 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca" pod "59a325ac-8170-44a7-a768-a4ca86e5890d" (UID: "59a325ac-8170-44a7-a768-a4ca86e5890d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:00.944927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.944898 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a325ac-8170-44a7-a768-a4ca86e5890d-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:00.944927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:00.944927 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk7n\" (UniqueName: \"kubernetes.io/projected/59a325ac-8170-44a7-a768-a4ca86e5890d-kube-api-access-6dk7n\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:01.502271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:01.502244 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" Oct 29 22:32:01.502736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:01.502272 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc" event={"ID":"59a325ac-8170-44a7-a768-a4ca86e5890d","Type":"ContainerDied","Data":"c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3"} Oct 29 22:32:01.502736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:01.502319 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2a0ff8526891dfcaf319d466c9283052fdc500e8182ad6607101608d54058f3" Oct 29 22:32:08.037371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:08.037333 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72"] Oct 29 22:32:08.050105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:08.050065 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d168ad-h9v72"] Oct 29 22:32:08.389237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:08.389165 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb0868a-c24a-4380-b835-a9a36f40be7b" path="/var/lib/kubelet/pods/8fb0868a-c24a-4380-b835-a9a36f40be7b/volumes" Oct 29 22:32:09.752196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.752161 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:09.753204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753186 4543 cpu_manager.go:414] "containerMap: removing container" podUID="59a325ac-8170-44a7-a768-a4ca86e5890d" containerName="revoke-keys" Oct 29 22:32:09.753204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753204 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a325ac-8170-44a7-a768-a4ca86e5890d" containerName="revoke-keys" Oct 29 22:32:09.753342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753238 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ed2b9bf2-b313-4f1c-8e40-570049284e49" containerName="revoke-keys" Oct 29 22:32:09.753342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753243 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2b9bf2-b313-4f1c-8e40-570049284e49" containerName="revoke-keys" Oct 29 22:32:09.753436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753409 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a325ac-8170-44a7-a768-a4ca86e5890d" containerName="revoke-keys" Oct 29 22:32:09.753436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.753424 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2b9bf2-b313-4f1c-8e40-570049284e49" containerName="revoke-keys" Oct 29 22:32:09.758132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.758113 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.775640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.775608 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"agc-cred-d3185baddced\"" Oct 29 22:32:09.781170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.781142 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:09.783250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.783228 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"routing-overlay\"" Oct 29 22:32:09.783484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.783282 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-e2e--605ba260-dockercfg-56bp5\"" Oct 29 22:32:09.783484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.783425 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"openshift-service-ca.crt\"" Oct 29 22:32:09.783686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.783468 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"kube-root-ca.crt\"" Oct 29 22:32:09.783686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.783531 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9ed94\"" Oct 29 22:32:09.878644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.878619 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.878840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.878819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.878890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.878861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4kg7\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.878953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.878939 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.980586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.980562 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.980709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.980695 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.980776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.980724 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4kg7\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.980834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.980773 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.981268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.981245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.981455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.981436 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:09.983712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:09.983676 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:10.006896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.006839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4kg7\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:10.100566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.100531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:10.152018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.144121 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r"] Oct 29 22:32:10.378090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.377695 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r"] Oct 29 22:32:10.379624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.379595 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.399787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.399762 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:32:10.399938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.399819 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:10.428200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.428173 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2b2\" (UniqueName: \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.428427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.428393 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.428802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.428772 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.530490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.530454 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.530685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.530634 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.530818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.530792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2b2\" (UniqueName: \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.531406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.531384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.533043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.533017 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.556432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.556410 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2b2\" (UniqueName: \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\") pod \"maas-api-revoke-sub-5f43767f-r8c8r\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:10.704437 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:10.704406 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaec3c81_b973_448c_835e_f0c799fdffce.slice/crio-647c6ca72f04a762c8bff0868a418a5d6c38530e9fb2d8e47689ea255036c126 WatchSource:0}: Error finding container 647c6ca72f04a762c8bff0868a418a5d6c38530e9fb2d8e47689ea255036c126: Status 404 returned error can't find the container with id 647c6ca72f04a762c8bff0868a418a5d6c38530e9fb2d8e47689ea255036c126 Oct 29 22:32:10.704668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.704647 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:10.764624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:10.761280 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:11.363775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.363747 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r"] Oct 29 22:32:11.364315 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:11.364292 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c632767_2bb2_4264_b04f_39661ce36edc.slice/crio-1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0 WatchSource:0}: Error finding container 1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0: Status 404 returned error can't find the container with id 1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0 Oct 29 22:32:11.832657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.832617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" event={"ID":"caec3c81-b973-448c-835e-f0c799fdffce","Type":"ContainerStarted","Data":"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884"} Oct 29 22:32:11.836725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.836700 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:11.836924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.836908 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" event={"ID":"caec3c81-b973-448c-835e-f0c799fdffce","Type":"ContainerStarted","Data":"647c6ca72f04a762c8bff0868a418a5d6c38530e9fb2d8e47689ea255036c126"} Oct 29 22:32:11.861760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.858270 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:11.878103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.871263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" event={"ID":"1c632767-2bb2-4264-b04f-39661ce36edc","Type":"ContainerStarted","Data":"1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0"} Oct 29 22:32:11.878103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.871387 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:11.886274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.886142 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:11.913100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.899336 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"openshift-service-ca.crt\"" Oct 29 22:32:11.960105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.959720 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb"] Oct 29 22:32:11.989091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.976551 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"routing-overlay\"" Oct 29 22:32:11.989091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.976841 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"kube-root-ca.crt\"" Oct 29 22:32:11.989091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.976947 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-e2e--b7017604-dockercfg-gkpdl\"" Oct 29 22:32:11.989091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.977005 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-2490cdadee51\"" Oct 29 22:32:11.989091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:11.987183 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:32:12.033105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.027570 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0412c3\"/\"agc-cred-31abe10e24c1\"" Oct 29 22:32:12.111650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.088704 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.118105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.116383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.118105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.116504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.118105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.116672 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfj6p\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.134728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.133979 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb"] Oct 29 22:32:12.134728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.134123 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.161575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.159997 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" podStartSLOduration=3.15998139 podStartE2EDuration="3.15998139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:11.986796283 +0000 UTC m=+2993.728726957" watchObservedRunningTime="2026-10-29 22:32:12.15998139 +0000 UTC m=+2993.901912064" Oct 29 22:32:12.223006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.222972 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.223257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htmk\" (UniqueName: \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.223257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223143 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.223257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223188 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.223439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223319 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.223439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223403 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.223553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.223532 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfj6p\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.224879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.224853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.225042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.224853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.226447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.226427 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.250949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.250925 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfj6p\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\") pod \"payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.325102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.325044 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htmk\" (UniqueName: \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.325207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.325159 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.325450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.325433 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.326055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.326037 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.327633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.327611 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.350970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.350947 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htmk\" (UniqueName: \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\") pod \"maas-api-revoke-sub-0b668fb5-7vztb\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.467746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.467723 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:12.517273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.517244 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:12.784216 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:12.784184 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74e3de8_5c24_4b4e_b646_4a84daba99b5.slice/crio-0e44b4fdb355700b26a0934adab50c3c61eec3950b0e233862204731b4427323 WatchSource:0}: Error finding container 0e44b4fdb355700b26a0934adab50c3c61eec3950b0e233862204731b4427323: Status 404 returned error can't find the container with id 0e44b4fdb355700b26a0934adab50c3c61eec3950b0e233862204731b4427323 Oct 29 22:32:12.784921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.784898 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:12.866220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.866192 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb"] Oct 29 22:32:12.874583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.874551 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" event={"ID":"c74e3de8-5c24-4b4e-b646-4a84daba99b5","Type":"ContainerStarted","Data":"0e44b4fdb355700b26a0934adab50c3c61eec3950b0e233862204731b4427323"} Oct 29 22:32:12.876495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:12.876472 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" event={"ID":"1c632767-2bb2-4264-b04f-39661ce36edc","Type":"ContainerStarted","Data":"a022030bd99bae4bb0f42f66e5620b4fe6ade5c8aa14ef1607c7fc82924ab428"} Oct 29 22:32:12.920547 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:12.920509 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4a6fab_33ff_4d33_80ba_8e37f4a95cd4.slice/crio-899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f WatchSource:0}: Error finding container 899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f: Status 404 returned error can't find the container with id 899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f Oct 29 22:32:13.888117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.888066 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" event={"ID":"c74e3de8-5c24-4b4e-b646-4a84daba99b5","Type":"ContainerStarted","Data":"6b5419c14a3a9fb7ca9b1bd0939db5e5639b423435d0b8238baa18386e8b8367"} Oct 29 22:32:13.888575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.888157 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:13.889836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.889814 4543 generic.go:356] "Generic (PLEG): container finished" podID="1c632767-2bb2-4264-b04f-39661ce36edc" containerID="a022030bd99bae4bb0f42f66e5620b4fe6ade5c8aa14ef1607c7fc82924ab428" exitCode=0 Oct 29 22:32:13.889938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.889900 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" event={"ID":"1c632767-2bb2-4264-b04f-39661ce36edc","Type":"ContainerDied","Data":"a022030bd99bae4bb0f42f66e5620b4fe6ade5c8aa14ef1607c7fc82924ab428"} Oct 29 22:32:13.891854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.891828 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" event={"ID":"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4","Type":"ContainerStarted","Data":"76a5b5839c60eaff5dec858aba579d1dba6522ff8a12597de254bab1368dc170"} Oct 29 22:32:13.891972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.891887 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" event={"ID":"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4","Type":"ContainerStarted","Data":"899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f"} Oct 29 22:32:13.941284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.941230 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" podStartSLOduration=2.941216527 podStartE2EDuration="2.941216527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:13.932290384 +0000 UTC m=+2995.674221058" watchObservedRunningTime="2026-10-29 22:32:13.941216527 +0000 UTC m=+2995.683147226" Oct 29 22:32:13.981063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:13.981011 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" podStartSLOduration=2.98099756 podStartE2EDuration="2.98099756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:13.975200427 +0000 UTC m=+2995.717131101" watchObservedRunningTime="2026-10-29 22:32:13.98099756 +0000 UTC m=+2995.722928232" Oct 29 22:32:14.223779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.223754 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:14.281656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.278935 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\") pod \"1c632767-2bb2-4264-b04f-39661ce36edc\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " Oct 29 22:32:14.281656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.279251 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\" (UniqueName: \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\") pod \"1c632767-2bb2-4264-b04f-39661ce36edc\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " Oct 29 22:32:14.281656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.279311 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca" pod "1c632767-2bb2-4264-b04f-39661ce36edc" (UID: "1c632767-2bb2-4264-b04f-39661ce36edc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:14.281656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.279316 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\") pod \"1c632767-2bb2-4264-b04f-39661ce36edc\" (UID: \"1c632767-2bb2-4264-b04f-39661ce36edc\") " Oct 29 22:32:14.281656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.280296 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:14.282448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.282412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca" pod "1c632767-2bb2-4264-b04f-39661ce36edc" (UID: "1c632767-2bb2-4264-b04f-39661ce36edc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:14.289701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.289679 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2" pod "1c632767-2bb2-4264-b04f-39661ce36edc" (UID: "1c632767-2bb2-4264-b04f-39661ce36edc"). InnerVolumeSpecName "kube-api-access-xj2b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:14.382361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.382336 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2b2\" (UniqueName: \"kubernetes.io/projected/1c632767-2bb2-4264-b04f-39661ce36edc-kube-api-access-xj2b2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:14.382361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.382362 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c632767-2bb2-4264-b04f-39661ce36edc-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:14.903328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.903291 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" Oct 29 22:32:14.903812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.903285 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r" event={"ID":"1c632767-2bb2-4264-b04f-39661ce36edc","Type":"ContainerDied","Data":"1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0"} Oct 29 22:32:14.903812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.903470 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a3ce967690ffc4ec88d574f06860f298683478a635383213ece5f87342053d0" Oct 29 22:32:14.908299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.907134 4543 generic.go:356] "Generic (PLEG): container finished" podID="8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" containerID="76a5b5839c60eaff5dec858aba579d1dba6522ff8a12597de254bab1368dc170" exitCode=0 Oct 29 22:32:14.908299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:14.908169 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" event={"ID":"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4","Type":"ContainerDied","Data":"76a5b5839c60eaff5dec858aba579d1dba6522ff8a12597de254bab1368dc170"} Oct 29 22:32:16.093883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.093856 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:16.232124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.232093 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\") pod \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " Oct 29 22:32:16.232288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.232215 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\") pod \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " Oct 29 22:32:16.232407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.232393 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\" (UniqueName: \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\") pod \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\" (UID: \"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4\") " Oct 29 22:32:16.232482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.232429 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca" pod "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" (UID: "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:16.233098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.233045 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:16.234450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.234422 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca" pod "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" (UID: "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:16.234593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.234574 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk" pod "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" (UID: "8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4"). InnerVolumeSpecName "kube-api-access-8htmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:16.334642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.334607 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htmk\" (UniqueName: \"kubernetes.io/projected/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-kube-api-access-8htmk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:16.334642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:16.334641 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:17.055536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:17.044846 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" event={"ID":"8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4","Type":"ContainerDied","Data":"899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f"} Oct 29 22:32:17.055536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:17.044911 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="899ec7804f2a8ca68b1d71d69c2bc9d0a1c0142ac2b927566fca745f9e624f6f" Oct 29 22:32:17.055536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:17.044979 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb" Oct 29 22:32:20.774585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.774545 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v"] Oct 29 22:32:20.775775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.775748 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" containerName="revoke-keys" Oct 29 22:32:20.775775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.775774 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" containerName="revoke-keys" Oct 29 22:32:20.776007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.775883 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1c632767-2bb2-4264-b04f-39661ce36edc" containerName="revoke-keys" Oct 29 22:32:20.776007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.775892 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c632767-2bb2-4264-b04f-39661ce36edc" containerName="revoke-keys" Oct 29 22:32:20.776195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.776178 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c632767-2bb2-4264-b04f-39661ce36edc" containerName="revoke-keys" Oct 29 22:32:20.776263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.776205 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" containerName="revoke-keys" Oct 29 22:32:20.780234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.780212 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:20.808593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.808490 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:32:20.809012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.808930 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:32:20.811954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.811934 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v"] Oct 29 22:32:20.898283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.898258 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkgw\" (UniqueName: \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:20.898521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.898497 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:20.898618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:20.898602 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.000335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.000309 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkgw\" (UniqueName: \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.000476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.000462 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.000534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.000514 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.001165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.001144 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.003055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.003035 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.026668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.026622 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkgw\" (UniqueName: \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\") pod \"maas-api-revoke-sub-0b2c35b1-j9v8v\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.108363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.108338 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:21.256953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.256924 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s"] Oct 29 22:32:21.262371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.262355 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.285840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.285772 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s"] Oct 29 22:32:21.347927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.347888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.348130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.347963 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.348130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.348013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw72g\" (UniqueName: \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.430531 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:21.430498 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9f1461_9a1e_4d94_ae46_f7c7d420f0d1.slice/crio-984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29 WatchSource:0}: Error finding container 984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29: Status 404 returned error can't find the container with id 984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29 Oct 29 22:32:21.431456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.431382 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v"] Oct 29 22:32:21.449595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.449568 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.449710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.449629 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.449710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.449681 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw72g\" (UniqueName: \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.450348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.450324 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.452396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.452377 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.475704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.475685 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw72g\" (UniqueName: \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\") pod \"maas-api-revoke-sub-63b4f2bf-km26s\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.647528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.647475 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:21.960411 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:21.960384 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f906c9_bc06_4019_8fa9_0ae5d3ea2890.slice/crio-0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2 WatchSource:0}: Error finding container 0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2: Status 404 returned error can't find the container with id 0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2 Oct 29 22:32:21.960940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:21.960870 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s"] Oct 29 22:32:22.092835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.092797 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" event={"ID":"40f906c9-bc06-4019-8fa9-0ae5d3ea2890","Type":"ContainerStarted","Data":"0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2"} Oct 29 22:32:22.094634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.094607 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" event={"ID":"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1","Type":"ContainerStarted","Data":"c8dc343555b7f8978d280f2e76ae102c2bc35b661e50a4d3539062526b3ac0c5"} Oct 29 22:32:22.094745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.094657 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" event={"ID":"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1","Type":"ContainerStarted","Data":"984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29"} Oct 29 22:32:22.139302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.139249 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" podStartSLOduration=2.139232034 podStartE2EDuration="2.139232034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:22.138367714 +0000 UTC m=+3003.880298388" watchObservedRunningTime="2026-10-29 22:32:22.139232034 +0000 UTC m=+3003.881162710" Oct 29 22:32:22.646653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.646624 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8"] Oct 29 22:32:22.652392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.652372 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.672764 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.672734 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8"] Oct 29 22:32:22.677291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.677265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzwp\" (UniqueName: \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.677521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.677498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.677667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.677647 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.780973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.780943 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzwp\" (UniqueName: \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.781172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.781138 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.781259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.781246 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.781939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.781920 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.783654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.783631 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.808647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.808620 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzwp\" (UniqueName: \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\") pod \"maas-api-revoke-sub-3ab99c2c-96mk8\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:22.879977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.879919 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:22.976683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:22.974888 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:23.108790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.108759 4543 generic.go:356] "Generic (PLEG): container finished" podID="40f906c9-bc06-4019-8fa9-0ae5d3ea2890" containerID="1ef972edae1e1b936dc03be4304f162c911d966cfa84be9c7ee87b04c70e81c3" exitCode=0 Oct 29 22:32:23.109032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.108820 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" event={"ID":"40f906c9-bc06-4019-8fa9-0ae5d3ea2890","Type":"ContainerDied","Data":"1ef972edae1e1b936dc03be4304f162c911d966cfa84be9c7ee87b04c70e81c3"} Oct 29 22:32:23.110702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.110675 4543 generic.go:356] "Generic (PLEG): container finished" podID="cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" containerID="c8dc343555b7f8978d280f2e76ae102c2bc35b661e50a4d3539062526b3ac0c5" exitCode=0 Oct 29 22:32:23.110825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.110751 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" event={"ID":"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1","Type":"ContainerDied","Data":"c8dc343555b7f8978d280f2e76ae102c2bc35b661e50a4d3539062526b3ac0c5"} Oct 29 22:32:23.311782 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:23.311739 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c5b2f1_1687_4532_8d87_75bd22a06818.slice/crio-ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19 WatchSource:0}: Error finding container ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19: Status 404 returned error can't find the container with id ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19 Oct 29 22:32:23.312351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.312327 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8"] Oct 29 22:32:23.802064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:23.802028 4543 scope.go:122] "RemoveContainer" containerID="e41c56c7bb9f34aa4529af31faa373fbbd9d7ab7817c9c8a724f11cb048cab8e" Oct 29 22:32:24.122615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.122580 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" event={"ID":"83c5b2f1-1687-4532-8d87-75bd22a06818","Type":"ContainerStarted","Data":"d8e05d4076097dab1f022a17e1835a88e3ca378fee9a71f4a75c46456fc3f4e1"} Oct 29 22:32:24.123002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.122639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" event={"ID":"83c5b2f1-1687-4532-8d87-75bd22a06818","Type":"ContainerStarted","Data":"ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19"} Oct 29 22:32:24.166032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.165952 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" podStartSLOduration=2.165932997 podStartE2EDuration="2.165932997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:24.165796217 +0000 UTC m=+3005.907726891" watchObservedRunningTime="2026-10-29 22:32:24.165932997 +0000 UTC m=+3005.907863671" Oct 29 22:32:24.330962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.330930 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:24.394920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.394848 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:24.412303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.412271 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\") pod \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " Oct 29 22:32:24.412688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.412668 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\") pod \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " Oct 29 22:32:24.412688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.412672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca" pod "40f906c9-bc06-4019-8fa9-0ae5d3ea2890" (UID: "40f906c9-bc06-4019-8fa9-0ae5d3ea2890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:24.412781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.412713 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\" (UniqueName: \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\") pod \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\" (UID: \"40f906c9-bc06-4019-8fa9-0ae5d3ea2890\") " Oct 29 22:32:24.413614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.413588 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.415642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.415615 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca" pod "40f906c9-bc06-4019-8fa9-0ae5d3ea2890" (UID: "40f906c9-bc06-4019-8fa9-0ae5d3ea2890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:24.415752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.415663 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g" pod "40f906c9-bc06-4019-8fa9-0ae5d3ea2890" (UID: "40f906c9-bc06-4019-8fa9-0ae5d3ea2890"). InnerVolumeSpecName "kube-api-access-jw72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:24.546101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.542814 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\" (UniqueName: \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\") pod \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " Oct 29 22:32:24.546101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.542973 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\") pod \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " Oct 29 22:32:24.546101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.543013 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\") pod \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\" (UID: \"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1\") " Oct 29 22:32:24.546101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.544059 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.546101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.544111 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw72g\" (UniqueName: \"kubernetes.io/projected/40f906c9-bc06-4019-8fa9-0ae5d3ea2890-kube-api-access-jw72g\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.576667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.575290 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca" pod "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" (UID: "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:24.581514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.581476 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca" pod "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" (UID: "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:24.582052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.582014 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw" pod "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" (UID: "cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1"). InnerVolumeSpecName "kube-api-access-drkgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:24.645861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.645783 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkgw\" (UniqueName: \"kubernetes.io/projected/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-kube-api-access-drkgw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.645861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.645823 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.645861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.645841 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:24.668396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.668358 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:32:24.911163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:24.911070 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:25.133879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.133846 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" Oct 29 22:32:25.133879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.133872 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s" event={"ID":"40f906c9-bc06-4019-8fa9-0ae5d3ea2890","Type":"ContainerDied","Data":"0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2"} Oct 29 22:32:25.134346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.133919 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fed6237eb032d8c4c534e28c435afcacd04fbe20717186560439f7b223f0dc2" Oct 29 22:32:25.135839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.135814 4543 generic.go:356] "Generic (PLEG): container finished" podID="83c5b2f1-1687-4532-8d87-75bd22a06818" containerID="d8e05d4076097dab1f022a17e1835a88e3ca378fee9a71f4a75c46456fc3f4e1" exitCode=0 Oct 29 22:32:25.135970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.135899 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" event={"ID":"83c5b2f1-1687-4532-8d87-75bd22a06818","Type":"ContainerDied","Data":"d8e05d4076097dab1f022a17e1835a88e3ca378fee9a71f4a75c46456fc3f4e1"} Oct 29 22:32:25.137351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.137331 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" Oct 29 22:32:25.137487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.137334 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v" event={"ID":"cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1","Type":"ContainerDied","Data":"984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29"} Oct 29 22:32:25.137487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.137456 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984f673312c5187d345d027daac0801a305cd8b6a20e93b0a33945990ba5ee29" Oct 29 22:32:25.561612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:25.561532 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:32:26.403514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.403448 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:26.489112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.489054 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\") pod \"83c5b2f1-1687-4532-8d87-75bd22a06818\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " Oct 29 22:32:26.489282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.489216 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\" (UniqueName: \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\") pod \"83c5b2f1-1687-4532-8d87-75bd22a06818\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " Oct 29 22:32:26.489368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.489299 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\") pod \"83c5b2f1-1687-4532-8d87-75bd22a06818\" (UID: \"83c5b2f1-1687-4532-8d87-75bd22a06818\") " Oct 29 22:32:26.489457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.489433 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca" pod "83c5b2f1-1687-4532-8d87-75bd22a06818" (UID: "83c5b2f1-1687-4532-8d87-75bd22a06818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:26.490243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.490218 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:26.491439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.491415 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca" pod "83c5b2f1-1687-4532-8d87-75bd22a06818" (UID: "83c5b2f1-1687-4532-8d87-75bd22a06818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:32:26.491523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.491459 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp" pod "83c5b2f1-1687-4532-8d87-75bd22a06818" (UID: "83c5b2f1-1687-4532-8d87-75bd22a06818"). InnerVolumeSpecName "kube-api-access-rhzwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:26.592009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.591987 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzwp\" (UniqueName: \"kubernetes.io/projected/83c5b2f1-1687-4532-8d87-75bd22a06818-kube-api-access-rhzwp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:26.592009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:26.592010 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c5b2f1-1687-4532-8d87-75bd22a06818-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:27.160867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:27.160842 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" Oct 29 22:32:27.161044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:27.160872 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8" event={"ID":"83c5b2f1-1687-4532-8d87-75bd22a06818","Type":"ContainerDied","Data":"ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19"} Oct 29 22:32:27.161044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:27.160922 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee5c8b62f5e8bc4684cd419ad9da0437dfdf65f672c306d969c996d16ba31b19" Oct 29 22:32:27.969691 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:32:27.969649 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:32:29.669472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:29.669384 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" podUID="78b613a2-9318-44e9-9d74-ab7780385006" containerName="payload-processing" containerID="cri-o://6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918" gracePeriod=25 Oct 29 22:32:29.984427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:29.984400 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:32:30.146789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.146734 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\") pod \"78b613a2-9318-44e9-9d74-ab7780385006\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " Oct 29 22:32:30.147165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.147004 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\") pod \"78b613a2-9318-44e9-9d74-ab7780385006\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " Oct 29 22:32:30.147165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.147115 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\") pod \"78b613a2-9318-44e9-9d74-ab7780385006\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " Oct 29 22:32:30.147332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.147178 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\") pod \"78b613a2-9318-44e9-9d74-ab7780385006\" (UID: \"78b613a2-9318-44e9-9d74-ab7780385006\") " Oct 29 22:32:30.149546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.149501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume" pod "78b613a2-9318-44e9-9d74-ab7780385006" (UID: "78b613a2-9318-44e9-9d74-ab7780385006"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.150122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.149878 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume" pod "78b613a2-9318-44e9-9d74-ab7780385006" (UID: "78b613a2-9318-44e9-9d74-ab7780385006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.152279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.152254 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials" pod "78b613a2-9318-44e9-9d74-ab7780385006" (UID: "78b613a2-9318-44e9-9d74-ab7780385006"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.152375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.152292 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz" pod "78b613a2-9318-44e9-9d74-ab7780385006" (UID: "78b613a2-9318-44e9-9d74-ab7780385006"). InnerVolumeSpecName "kube-api-access-2glrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.191934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.191902 4543 generic.go:356] "Generic (PLEG): container finished" podID="78b613a2-9318-44e9-9d74-ab7780385006" containerID="6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918" exitCode=0 Oct 29 22:32:30.192111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.191962 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" Oct 29 22:32:30.192111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.191982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" event={"ID":"78b613a2-9318-44e9-9d74-ab7780385006","Type":"ContainerDied","Data":"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918"} Oct 29 22:32:30.192111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.192045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt" event={"ID":"78b613a2-9318-44e9-9d74-ab7780385006","Type":"ContainerDied","Data":"a4a3250ccad953ffee3ae394879ee6f75934bed4fa79b35f9c9f215769a98080"} Oct 29 22:32:30.192253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.192090 4543 scope.go:122] "RemoveContainer" containerID="6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918" Oct 29 22:32:30.204631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.204610 4543 scope.go:122] "RemoveContainer" containerID="6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918" Oct 29 22:32:30.204912 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:32:30.204892 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918\": container with ID starting with 6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918 not found: ID does not exist" containerID="6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918" Oct 29 22:32:30.204978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.204924 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918"} err="failed to get container status \"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918\": rpc error: code = NotFound desc = could not find container \"6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918\": container with ID starting with 6d41063344f5106476854fa11032d9080e3c4217ab48bdfea9fbfd0c766fa918 not found: ID does not exist" Oct 29 22:32:30.251652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.251624 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.251652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.251653 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glrz\" (UniqueName: \"kubernetes.io/projected/78b613a2-9318-44e9-9d74-ab7780385006-kube-api-access-2glrz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.251827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.251669 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.251827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.251681 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78b613a2-9318-44e9-9d74-ab7780385006-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.253892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.253872 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:32:30.259436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.259403 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-5f5fjqfmt"] Oct 29 22:32:30.392651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.392622 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b613a2-9318-44e9-9d74-ab7780385006" path="/var/lib/kubelet/pods/78b613a2-9318-44e9-9d74-ab7780385006/volumes" Oct 29 22:32:30.562261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.562194 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" podUID="c9222c51-1bb6-4197-8d07-a6099b432436" containerName="payload-processing" containerID="cri-o://33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7" gracePeriod=25 Oct 29 22:32:30.871169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.871144 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:32:30.966951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.966921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\") pod \"c9222c51-1bb6-4197-8d07-a6099b432436\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " Oct 29 22:32:30.967139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.967069 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\") pod \"c9222c51-1bb6-4197-8d07-a6099b432436\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " Oct 29 22:32:30.967279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.967262 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\") pod \"c9222c51-1bb6-4197-8d07-a6099b432436\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " Oct 29 22:32:30.967422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.967404 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\") pod \"c9222c51-1bb6-4197-8d07-a6099b432436\" (UID: \"c9222c51-1bb6-4197-8d07-a6099b432436\") " Oct 29 22:32:30.967422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.967410 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume" pod "c9222c51-1bb6-4197-8d07-a6099b432436" (UID: "c9222c51-1bb6-4197-8d07-a6099b432436"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.967823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.967797 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume" pod "c9222c51-1bb6-4197-8d07-a6099b432436" (UID: "c9222c51-1bb6-4197-8d07-a6099b432436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:30.968535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.968508 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.968641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.968539 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9222c51-1bb6-4197-8d07-a6099b432436-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:30.969618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.969590 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb" pod "c9222c51-1bb6-4197-8d07-a6099b432436" (UID: "c9222c51-1bb6-4197-8d07-a6099b432436"). InnerVolumeSpecName "kube-api-access-mzzsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:30.969791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:30.969767 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials" pod "c9222c51-1bb6-4197-8d07-a6099b432436" (UID: "c9222c51-1bb6-4197-8d07-a6099b432436"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:31.069855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.069827 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:31.069855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.069853 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzzsb\" (UniqueName: \"kubernetes.io/projected/c9222c51-1bb6-4197-8d07-a6099b432436-kube-api-access-mzzsb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:31.205866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.205775 4543 generic.go:356] "Generic (PLEG): container finished" podID="c9222c51-1bb6-4197-8d07-a6099b432436" containerID="33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7" exitCode=0 Oct 29 22:32:31.205866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.205824 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" Oct 29 22:32:31.206201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.205860 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" event={"ID":"c9222c51-1bb6-4197-8d07-a6099b432436","Type":"ContainerDied","Data":"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7"} Oct 29 22:32:31.206201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.205914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf" event={"ID":"c9222c51-1bb6-4197-8d07-a6099b432436","Type":"ContainerDied","Data":"b461817f8478961777a74f942792e062ad8a8a807cb40c7647e78209d4085fd4"} Oct 29 22:32:31.206201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.205936 4543 scope.go:122] "RemoveContainer" containerID="33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7" Oct 29 22:32:31.225005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.224984 4543 scope.go:122] "RemoveContainer" containerID="33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7" Oct 29 22:32:31.226472 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:32:31.225679 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7\": container with ID starting with 33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7 not found: ID does not exist" containerID="33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7" Oct 29 22:32:31.226472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.225714 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7"} err="failed to get container status \"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7\": rpc error: code = NotFound desc = could not find container \"33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7\": container with ID starting with 33636539caa32a85079054bb26b6695846c63ee445832af9c2daefa21ad762d7 not found: ID does not exist" Oct 29 22:32:31.270438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.270415 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:32:31.278395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:31.278371 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e7b56d/payload-processing-external-model-e2e-extmod-b-e7b56d-85dbms5rf"] Oct 29 22:32:32.389104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:32.389060 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9222c51-1bb6-4197-8d07-a6099b432436" path="/var/lib/kubelet/pods/c9222c51-1bb6-4197-8d07-a6099b432436/volumes" Oct 29 22:32:33.575804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:33.575772 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:38.576903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:38.576863 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" podUID="caec3c81-b973-448c-835e-f0c799fdffce" containerName="payload-processing" containerID="cri-o://0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884" gracePeriod=25 Oct 29 22:32:38.983668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:38.983269 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:39.016098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.015298 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\") pod \"caec3c81-b973-448c-835e-f0c799fdffce\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " Oct 29 22:32:39.016098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.015351 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\") pod \"caec3c81-b973-448c-835e-f0c799fdffce\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " Oct 29 22:32:39.016098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.015455 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\") pod \"caec3c81-b973-448c-835e-f0c799fdffce\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " Oct 29 22:32:39.016098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.015507 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\") pod \"caec3c81-b973-448c-835e-f0c799fdffce\" (UID: \"caec3c81-b973-448c-835e-f0c799fdffce\") " Oct 29 22:32:39.017347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.017315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume" pod "caec3c81-b973-448c-835e-f0c799fdffce" (UID: "caec3c81-b973-448c-835e-f0c799fdffce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:39.039716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.039682 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume" pod "caec3c81-b973-448c-835e-f0c799fdffce" (UID: "caec3c81-b973-448c-835e-f0c799fdffce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:39.050123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.040596 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials" pod "caec3c81-b973-448c-835e-f0c799fdffce" (UID: "caec3c81-b973-448c-835e-f0c799fdffce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:39.084063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.084029 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7" pod "caec3c81-b973-448c-835e-f0c799fdffce" (UID: "caec3c81-b973-448c-835e-f0c799fdffce"). InnerVolumeSpecName "kube-api-access-s4kg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:39.120682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.120593 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:39.120924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.120903 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:39.121033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.121019 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4kg7\" (UniqueName: \"kubernetes.io/projected/caec3c81-b973-448c-835e-f0c799fdffce-kube-api-access-s4kg7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:39.147101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.129231 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/caec3c81-b973-448c-835e-f0c799fdffce-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:39.573215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.573177 4543 generic.go:356] "Generic (PLEG): container finished" podID="caec3c81-b973-448c-835e-f0c799fdffce" containerID="0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884" exitCode=0 Oct 29 22:32:39.573429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.573229 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" Oct 29 22:32:39.573429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.573266 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" event={"ID":"caec3c81-b973-448c-835e-f0c799fdffce","Type":"ContainerDied","Data":"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884"} Oct 29 22:32:39.573429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.573354 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88" event={"ID":"caec3c81-b973-448c-835e-f0c799fdffce","Type":"ContainerDied","Data":"647c6ca72f04a762c8bff0868a418a5d6c38530e9fb2d8e47689ea255036c126"} Oct 29 22:32:39.573429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.573383 4543 scope.go:122] "RemoveContainer" containerID="0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884" Oct 29 22:32:39.585999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.585834 4543 scope.go:122] "RemoveContainer" containerID="0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884" Oct 29 22:32:39.586287 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:32:39.586113 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884\": container with ID starting with 0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884 not found: ID does not exist" containerID="0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884" Oct 29 22:32:39.586287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.586143 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884"} err="failed to get container status \"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884\": rpc error: code = NotFound desc = could not find container \"0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884\": container with ID starting with 0f1b986309b5dc4f2887f48e6edfb7c6a7b79783fdbf5c414f154d2314654884 not found: ID does not exist" Oct 29 22:32:39.637606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.637581 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:39.650607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:39.650586 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-9f5bsgc88"] Oct 29 22:32:40.391029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:40.390996 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caec3c81-b973-448c-835e-f0c799fdffce" path="/var/lib/kubelet/pods/caec3c81-b973-448c-835e-f0c799fdffce/volumes" Oct 29 22:32:40.510884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:40.510856 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:45.511151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.511108 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" podUID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" containerName="payload-processing" containerID="cri-o://6b5419c14a3a9fb7ca9b1bd0939db5e5639b423435d0b8238baa18386e8b8367" gracePeriod=25 Oct 29 22:32:45.638025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.637998 4543 generic.go:356] "Generic (PLEG): container finished" podID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" containerID="6b5419c14a3a9fb7ca9b1bd0939db5e5639b423435d0b8238baa18386e8b8367" exitCode=0 Oct 29 22:32:45.638160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.638108 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" event={"ID":"c74e3de8-5c24-4b4e-b646-4a84daba99b5","Type":"ContainerDied","Data":"6b5419c14a3a9fb7ca9b1bd0939db5e5639b423435d0b8238baa18386e8b8367"} Oct 29 22:32:45.806782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.806757 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:45.863286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.863259 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\") pod \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " Oct 29 22:32:45.863441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.863401 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\") pod \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " Oct 29 22:32:45.863441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.863427 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\") pod \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " Oct 29 22:32:45.863655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.863635 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\") pod \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\" (UID: \"c74e3de8-5c24-4b4e-b646-4a84daba99b5\") " Oct 29 22:32:45.863856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.863831 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume" pod "c74e3de8-5c24-4b4e-b646-4a84daba99b5" (UID: "c74e3de8-5c24-4b4e-b646-4a84daba99b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:45.864089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.864021 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume" pod "c74e3de8-5c24-4b4e-b646-4a84daba99b5" (UID: "c74e3de8-5c24-4b4e-b646-4a84daba99b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:45.864900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.864874 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:45.864995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.864906 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e3de8-5c24-4b4e-b646-4a84daba99b5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:45.866007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.865977 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials" pod "c74e3de8-5c24-4b4e-b646-4a84daba99b5" (UID: "c74e3de8-5c24-4b4e-b646-4a84daba99b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:45.866242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.866218 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p" pod "c74e3de8-5c24-4b4e-b646-4a84daba99b5" (UID: "c74e3de8-5c24-4b4e-b646-4a84daba99b5"). InnerVolumeSpecName "kube-api-access-tfj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:45.967108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.967064 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:45.967108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:45.967106 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfj6p\" (UniqueName: \"kubernetes.io/projected/c74e3de8-5c24-4b4e-b646-4a84daba99b5-kube-api-access-tfj6p\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:32:46.714263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:46.714216 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" event={"ID":"c74e3de8-5c24-4b4e-b646-4a84daba99b5","Type":"ContainerDied","Data":"0e44b4fdb355700b26a0934adab50c3c61eec3950b0e233862204731b4427323"} Oct 29 22:32:46.715201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:46.715180 4543 scope.go:122] "RemoveContainer" containerID="6b5419c14a3a9fb7ca9b1bd0939db5e5639b423435d0b8238baa18386e8b8367" Oct 29 22:32:46.715506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:46.715107 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q" Oct 29 22:32:46.807898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:46.803675 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:46.813500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:46.813472 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0412c3/payload-processing-external-model-e2e-extmod-a-0412c3-86dcbh68q"] Oct 29 22:32:48.338170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.338140 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:32:48.339296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339275 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" containerName="payload-processing" Oct 29 22:32:48.339296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339296 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339313 4543 cpu_manager.go:414] "containerMap: removing container" podUID="78b613a2-9318-44e9-9d74-ab7780385006" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339318 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b613a2-9318-44e9-9d74-ab7780385006" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339327 4543 cpu_manager.go:414] "containerMap: removing container" podUID="caec3c81-b973-448c-835e-f0c799fdffce" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339336 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caec3c81-b973-448c-835e-f0c799fdffce" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339380 4543 cpu_manager.go:414] "containerMap: removing container" podUID="83c5b2f1-1687-4532-8d87-75bd22a06818" containerName="revoke-keys" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339390 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c5b2f1-1687-4532-8d87-75bd22a06818" containerName="revoke-keys" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339446 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" containerName="revoke-keys" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339455 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" containerName="revoke-keys" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339474 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c9222c51-1bb6-4197-8d07-a6099b432436" containerName="payload-processing" Oct 29 22:32:48.339478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339479 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9222c51-1bb6-4197-8d07-a6099b432436" containerName="payload-processing" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339485 4543 cpu_manager.go:414] "containerMap: removing container" podUID="40f906c9-bc06-4019-8fa9-0ae5d3ea2890" containerName="revoke-keys" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339489 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f906c9-bc06-4019-8fa9-0ae5d3ea2890" containerName="revoke-keys" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339701 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" containerName="revoke-keys" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339722 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9222c51-1bb6-4197-8d07-a6099b432436" containerName="payload-processing" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339752 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f906c9-bc06-4019-8fa9-0ae5d3ea2890" containerName="revoke-keys" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339768 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" containerName="payload-processing" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339778 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b613a2-9318-44e9-9d74-ab7780385006" containerName="payload-processing" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339789 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="caec3c81-b973-448c-835e-f0c799fdffce" containerName="payload-processing" Oct 29 22:32:48.339896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.339800 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c5b2f1-1687-4532-8d87-75bd22a06818" containerName="revoke-keys" Oct 29 22:32:48.344401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.344384 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.364083 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.364045 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:32:48.368872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.368668 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0cb13f\"" Oct 29 22:32:48.368872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.368830 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0cb13f-dockercfg-p254z\"" Oct 29 22:32:48.404796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.404761 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74e3de8-5c24-4b4e-b646-4a84daba99b5" path="/var/lib/kubelet/pods/c74e3de8-5c24-4b4e-b646-4a84daba99b5/volumes" Oct 29 22:32:48.412201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.412172 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:32:48.422411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.419785 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.754163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.667783 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.754163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.667898 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8r4\" (UniqueName: \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.754163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.667942 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.754163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.668111 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pd6\" (UniqueName: \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.817101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.812167 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:32:48.824500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.824381 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pd6\" (UniqueName: \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.839102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.824782 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.839102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.824905 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8r4\" (UniqueName: \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.839102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.824937 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.839102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.832322 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.839102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.838285 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.868871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.868838 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pd6\" (UniqueName: \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\") pod \"payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:48.869580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.869554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8r4\" (UniqueName: \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\") pod \"payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:48.962770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:48.962737 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:49.140404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.138478 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:49.293431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.293400 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:32:49.295179 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:49.295151 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266ee6eb_04ae_4db4_9bb1_1e0e4d6406c7.slice/crio-97888b3122fe0a082e5fd1d25217b30646f7d8a98fde552c4e7f67a98c2391a7 WatchSource:0}: Error finding container 97888b3122fe0a082e5fd1d25217b30646f7d8a98fde552c4e7f67a98c2391a7: Status 404 returned error can't find the container with id 97888b3122fe0a082e5fd1d25217b30646f7d8a98fde552c4e7f67a98c2391a7 Oct 29 22:32:49.470460 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:49.470428 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf608efa0_2cf1_4c5c_b15f_7230fddac846.slice/crio-3b384e028826ef785035231071bc8e032da7c55ccc7c392fb362360385f7eaa0 WatchSource:0}: Error finding container 3b384e028826ef785035231071bc8e032da7c55ccc7c392fb362360385f7eaa0: Status 404 returned error can't find the container with id 3b384e028826ef785035231071bc8e032da7c55ccc7c392fb362360385f7eaa0 Oct 29 22:32:49.471369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.471346 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:32:49.856435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.856340 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" event={"ID":"f608efa0-2cf1-4c5c-b15f-7230fddac846","Type":"ContainerStarted","Data":"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c"} Oct 29 22:32:49.856435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.856409 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:32:49.856435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.856432 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" event={"ID":"f608efa0-2cf1-4c5c-b15f-7230fddac846","Type":"ContainerStarted","Data":"3b384e028826ef785035231071bc8e032da7c55ccc7c392fb362360385f7eaa0"} Oct 29 22:32:49.858126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.858098 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" event={"ID":"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7","Type":"ContainerStarted","Data":"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037"} Oct 29 22:32:49.858283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.858144 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" event={"ID":"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7","Type":"ContainerStarted","Data":"97888b3122fe0a082e5fd1d25217b30646f7d8a98fde552c4e7f67a98c2391a7"} Oct 29 22:32:49.858283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.858183 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:32:49.902303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.902256 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" podStartSLOduration=1.902243656 podStartE2EDuration="1.902243656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:49.89985257 +0000 UTC m=+3031.641783242" watchObservedRunningTime="2026-10-29 22:32:49.902243656 +0000 UTC m=+3031.644174515" Oct 29 22:32:49.945929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:49.945877 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" podStartSLOduration=1.9458646019999999 podStartE2EDuration="1.945864602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:49.943344343 +0000 UTC m=+3031.685275015" watchObservedRunningTime="2026-10-29 22:32:49.945864602 +0000 UTC m=+3031.687795276" Oct 29 22:32:52.044765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:52.044727 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl"] Oct 29 22:32:52.055223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:52.055194 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ecfee64-pw2nl"] Oct 29 22:32:52.389626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:52.389551 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad8b200-625c-4965-9290-287e6d25be96" path="/var/lib/kubelet/pods/dad8b200-625c-4965-9290-287e6d25be96/volumes" Oct 29 22:32:53.602421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.602209 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:32:53.824099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.821620 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:32:53.824099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.821760 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:53.886089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.885827 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-219ccc\"" Oct 29 22:32:53.886855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.886569 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-219ccc-dockercfg-8p47d\"" Oct 29 22:32:53.962549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.962517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrdq\" (UniqueName: \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:53.962715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:53.962556 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.064507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.064475 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrdq\" (UniqueName: \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.064507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.064511 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.065271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.065251 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.101418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.101393 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrdq\" (UniqueName: \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\") pod \"payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.186836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.186781 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:54.498597 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:54.498569 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac3f9e6_12af_4993_8a74_e57d3a428b85.slice/crio-0a339a80f911f2c64acf8082cf80b0c326b5c4c175c9c1775154a0b53417021e WatchSource:0}: Error finding container 0a339a80f911f2c64acf8082cf80b0c326b5c4c175c9c1775154a0b53417021e: Status 404 returned error can't find the container with id 0a339a80f911f2c64acf8082cf80b0c326b5c4c175c9c1775154a0b53417021e Oct 29 22:32:54.500340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.500305 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:32:54.971730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.971697 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" event={"ID":"5ac3f9e6-12af-4993-8a74-e57d3a428b85","Type":"ContainerStarted","Data":"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42"} Oct 29 22:32:54.972219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.971759 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" event={"ID":"5ac3f9e6-12af-4993-8a74-e57d3a428b85","Type":"ContainerStarted","Data":"0a339a80f911f2c64acf8082cf80b0c326b5c4c175c9c1775154a0b53417021e"} Oct 29 22:32:54.972219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:54.971789 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:32:55.023451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:55.023400 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" podStartSLOduration=2.023387443 podStartE2EDuration="2.023387443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:55.02196768 +0000 UTC m=+3036.763898365" watchObservedRunningTime="2026-10-29 22:32:55.023387443 +0000 UTC m=+3036.765318115" Oct 29 22:32:57.962424 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:32:57.962386 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:32:58.050164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.050132 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:32:58.056039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.056022 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.072606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.072580 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:32:58.079091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.079055 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52429a\"" Oct 29 22:32:58.079225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.079070 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52429a-dockercfg-lsxm4\"" Oct 29 22:32:58.256640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.256547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.256828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.256734 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jhj\" (UniqueName: \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.365379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.365062 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.365379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.365278 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jhj\" (UniqueName: \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.374355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.366694 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.488719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.488542 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jhj\" (UniqueName: \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\") pod \"payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.678533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.678441 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:32:58.783100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.756980 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:32:58.805708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.794939 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:58.805708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.801886 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:32:58.926344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.924482 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:58.926344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:58.924715 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458kp\" (UniqueName: \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.049395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.049169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.050138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.050107 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458kp\" (UniqueName: \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.051867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.051823 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.105485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.105416 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458kp\" (UniqueName: \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\") pod \"payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.369147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.360810 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:32:59.683656 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:32:59.661283 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7171c580_4a79_47a3_8662_69bfc3378505.slice/crio-4e2c66f582b4be6051d626b60ce9140ab65408632c5b1e2b8b0df0ce070728f8 WatchSource:0}: Error finding container 4e2c66f582b4be6051d626b60ce9140ab65408632c5b1e2b8b0df0ce070728f8: Status 404 returned error can't find the container with id 4e2c66f582b4be6051d626b60ce9140ab65408632c5b1e2b8b0df0ce070728f8 Oct 29 22:32:59.683656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:32:59.662973 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:33:00.094139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.076387 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:33:00.122180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.122042 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" event={"ID":"7171c580-4a79-47a3-8662-69bfc3378505","Type":"ContainerStarted","Data":"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd"} Oct 29 22:33:00.122507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.122482 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" event={"ID":"7171c580-4a79-47a3-8662-69bfc3378505","Type":"ContainerStarted","Data":"4e2c66f582b4be6051d626b60ce9140ab65408632c5b1e2b8b0df0ce070728f8"} Oct 29 22:33:00.122659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.122643 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:33:00.125330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.125228 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" event={"ID":"c43fb9b5-8654-4ed1-985e-8109c320e110","Type":"ContainerStarted","Data":"910a42f51fa77565854df2aab86e3d47032d1315fb779f302783e8a2e8728ee0"} Oct 29 22:33:00.209101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.205728 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" podStartSLOduration=2.205705106 podStartE2EDuration="2.205705106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:00.194758393 +0000 UTC m=+3041.936689068" watchObservedRunningTime="2026-10-29 22:33:00.205705106 +0000 UTC m=+3041.947635779" Oct 29 22:33:00.875155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.874736 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:33:00.876197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:00.876023 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:33:01.137138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:01.137022 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" event={"ID":"c43fb9b5-8654-4ed1-985e-8109c320e110","Type":"ContainerStarted","Data":"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a"} Oct 29 22:33:01.137551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:01.137402 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:33:01.185555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:01.185506 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" podStartSLOduration=3.185493053 podStartE2EDuration="3.185493053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:01.180884432 +0000 UTC m=+3042.922815106" watchObservedRunningTime="2026-10-29 22:33:01.185493053 +0000 UTC m=+3042.927423726" Oct 29 22:33:05.984697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:05.984658 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:33:07.734126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.734091 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:33:07.739795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.739777 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.764897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.764869 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:33:07.769574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.769545 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b9ed94-openshift-default-dockercfg-7pdjl\"" Oct 29 22:33:07.885679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885645 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.885853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.885853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885788 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.885853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885811 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.885980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.885980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.885940 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.886051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.886013 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.886200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.886186 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.886260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.886248 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjcl\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.987694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987618 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.987694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987664 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.987899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987786 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.987899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.987899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987880 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987939 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.987995 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988035 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988201 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988281 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988296 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjcl\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988354 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988747 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.988934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.988898 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.990245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.990220 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:07.990468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:07.990449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:08.020521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.020488 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:08.021749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.021730 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjcl\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\") pod \"e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:08.061588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.061562 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:08.408105 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:33:08.408049 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865040ec_fa04_46b4_9074_2dc97d5842e1.slice/crio-208e5b330f45b285c44696062d2d0ba5b6c634e719c0c6cf869622132a0d4f6e WatchSource:0}: Error finding container 208e5b330f45b285c44696062d2d0ba5b6c634e719c0c6cf869622132a0d4f6e: Status 404 returned error can't find the container with id 208e5b330f45b285c44696062d2d0ba5b6c634e719c0c6cf869622132a0d4f6e Oct 29 22:33:08.408228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.408195 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:33:08.411223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.411187 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:08.411327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.411273 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:08.411327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:08.411317 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:33:09.048999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:09.048969 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm"] Oct 29 22:33:09.061801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:09.061776 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab65700-phxvm"] Oct 29 22:33:09.242635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:09.242596 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" event={"ID":"865040ec-fa04-46b4-9074-2dc97d5842e1","Type":"ContainerStarted","Data":"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500"} Oct 29 22:33:09.242818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:09.242662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" event={"ID":"865040ec-fa04-46b4-9074-2dc97d5842e1","Type":"ContainerStarted","Data":"208e5b330f45b285c44696062d2d0ba5b6c634e719c0c6cf869622132a0d4f6e"} Oct 29 22:33:09.304433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:09.304332 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podStartSLOduration=2.3043162600000002 podStartE2EDuration="2.30431626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:09.30136096 +0000 UTC m=+3051.043291634" watchObservedRunningTime="2026-10-29 22:33:09.30431626 +0000 UTC m=+3051.046246934" Oct 29 22:33:10.061782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:10.061744 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:10.063329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:10.063299 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" start-of-body= Oct 29 22:33:10.063490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:10.063356 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" Oct 29 22:33:10.395121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:10.395013 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ccc2c8-a104-49e8-8793-19e087e8e368" path="/var/lib/kubelet/pods/94ccc2c8-a104-49e8-8793-19e087e8e368/volumes" Oct 29 22:33:11.062651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:11.062572 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" start-of-body= Oct 29 22:33:11.062994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:11.062641 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" Oct 29 22:33:11.156387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:11.156358 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:33:12.062658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:12.062622 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" start-of-body= Oct 29 22:33:12.063044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:12.062684 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" Oct 29 22:33:12.197389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:12.197362 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:33:13.062542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:13.062507 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" start-of-body= Oct 29 22:33:13.062721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:13.062574 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" Oct 29 22:33:14.067398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:14.067360 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" start-of-body= Oct 29 22:33:14.068015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:14.067976 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.154:15021/healthz/ready\": dial tcp 10.133.1.154:15021: connect: connection refused" Oct 29 22:33:15.109296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:15.109261 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:15.109968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:15.109732 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:15.110726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:15.110705 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:33:23.851816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:23.851781 4543 scope.go:122] "RemoveContainer" containerID="864d38fae0fd4a0182896e225783c2b0ea020020564bc9bf3afbcc0eb0dae8f1" Oct 29 22:33:23.864884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:23.864854 4543 scope.go:122] "RemoveContainer" containerID="1b5eb5943163102f914910e84cc385ff4aaa522c89eb67035f9c731a0271efab" Oct 29 22:33:23.879698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:33:23.879677 4543 scope.go:122] "RemoveContainer" containerID="4af5b83af1a49f38beba6b19373acd41cec88ce489c8f6a11e4d0b6c6ce37d67" Oct 29 22:33:27.962965 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:33:27.962921 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:33:57.963598 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:33:57.963563 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:34:04.778149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:04.777237 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 22:34:04.814227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:04.814166 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:04.935582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:04.935544 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Oct 29 22:34:04.980101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:04.973529 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:12.053122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:12.053061 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5"] Oct 29 22:34:12.063885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:12.063847 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959407d4-r2sx5"] Oct 29 22:34:12.392057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:12.391982 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08530db3-b66e-441b-bbfe-1a77037e2fd4" path="/var/lib/kubelet/pods/08530db3-b66e-441b-bbfe-1a77037e2fd4/volumes" Oct 29 22:34:24.016304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:24.016260 4543 scope.go:122] "RemoveContainer" containerID="8e434f29347737b2cec22886faaf2364717647737fa3a72b148b140b06beeda6" Oct 29 22:34:24.051670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:24.051644 4543 scope.go:122] "RemoveContainer" containerID="9b9bb27c033cec7f77d293cf84dcdde5973bd0c57823d2ba4821d81a47eb722d" Oct 29 22:34:27.963556 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:34:27.963519 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:34:46.366103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:46.365014 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 29 22:34:46.430099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:46.412158 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 29 22:34:46.727454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:46.727406 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l" podUID="e386a9e9-223d-4c2d-96c2-148d940e4044" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.776101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:46.766994 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.836950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:46.829495 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:51.482251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.482221 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq"] Oct 29 22:34:51.487690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.487672 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.507343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.507319 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq"] Oct 29 22:34:51.513622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.513596 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:34:51.513622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.513615 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:34:51.569688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.569665 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5j7\" (UniqueName: \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.569815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.569722 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.569918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.569899 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.671138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.671106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5j7\" (UniqueName: \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.671267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.671168 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.671267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.671260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.671771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.671750 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.673766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.673739 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.696984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.696963 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5j7\" (UniqueName: \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\") pod \"maas-api-revoke-sub-54c86ab4-xznpq\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:51.804375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:51.804327 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:52.126054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:52.126025 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq"] Oct 29 22:34:52.126193 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:34:52.126116 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f45e13_d301_4a71_ba7f_d4212992222a.slice/crio-402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e WatchSource:0}: Error finding container 402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e: Status 404 returned error can't find the container with id 402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e Oct 29 22:34:52.952155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:52.952064 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" event={"ID":"e3f45e13-d301-4a71-ba7f-d4212992222a","Type":"ContainerStarted","Data":"2415f582f2a17277187e109a39c1f320801f95d0fdc13a0c4d71f84a523e7d55"} Oct 29 22:34:52.952155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:52.952140 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" event={"ID":"e3f45e13-d301-4a71-ba7f-d4212992222a","Type":"ContainerStarted","Data":"402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e"} Oct 29 22:34:53.077483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:53.077417 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" podStartSLOduration=2.077399493 podStartE2EDuration="2.077399493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:53.001004177 +0000 UTC m=+3154.742934851" watchObservedRunningTime="2026-10-29 22:34:53.077399493 +0000 UTC m=+3154.819330166" Oct 29 22:34:53.963546 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:53.963510 4543 generic.go:356] "Generic (PLEG): container finished" podID="e3f45e13-d301-4a71-ba7f-d4212992222a" containerID="2415f582f2a17277187e109a39c1f320801f95d0fdc13a0c4d71f84a523e7d55" exitCode=0 Oct 29 22:34:53.964041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:53.963627 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" event={"ID":"e3f45e13-d301-4a71-ba7f-d4212992222a","Type":"ContainerDied","Data":"2415f582f2a17277187e109a39c1f320801f95d0fdc13a0c4d71f84a523e7d55"} Oct 29 22:34:55.163244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.163220 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:55.247143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.247058 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\") pod \"e3f45e13-d301-4a71-ba7f-d4212992222a\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " Oct 29 22:34:55.247310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.247213 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\" (UniqueName: \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\") pod \"e3f45e13-d301-4a71-ba7f-d4212992222a\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " Oct 29 22:34:55.247365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.247321 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\") pod \"e3f45e13-d301-4a71-ba7f-d4212992222a\" (UID: \"e3f45e13-d301-4a71-ba7f-d4212992222a\") " Oct 29 22:34:55.247679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.247654 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca" pod "e3f45e13-d301-4a71-ba7f-d4212992222a" (UID: "e3f45e13-d301-4a71-ba7f-d4212992222a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:55.248130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.248069 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:34:55.249655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.249627 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca" pod "e3f45e13-d301-4a71-ba7f-d4212992222a" (UID: "e3f45e13-d301-4a71-ba7f-d4212992222a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:55.250018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.249998 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7" pod "e3f45e13-d301-4a71-ba7f-d4212992222a" (UID: "e3f45e13-d301-4a71-ba7f-d4212992222a"). InnerVolumeSpecName "kube-api-access-9f5j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:55.349459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.349428 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f5j7\" (UniqueName: \"kubernetes.io/projected/e3f45e13-d301-4a71-ba7f-d4212992222a-kube-api-access-9f5j7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:34:55.349459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.349455 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f45e13-d301-4a71-ba7f-d4212992222a-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:34:55.991767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.991732 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" Oct 29 22:34:55.991767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.991751 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq" event={"ID":"e3f45e13-d301-4a71-ba7f-d4212992222a","Type":"ContainerDied","Data":"402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e"} Oct 29 22:34:55.992043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:34:55.991803 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402235c88daa8d3af7dd90f6bd39e16f46b7bf342c1b51aaed59cb57b10e897e" Oct 29 22:34:57.961938 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:34:57.961899 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:35:02.171822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.171792 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9"] Oct 29 22:35:02.172941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.172911 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e3f45e13-d301-4a71-ba7f-d4212992222a" containerName="revoke-keys" Oct 29 22:35:02.172941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.172942 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f45e13-d301-4a71-ba7f-d4212992222a" containerName="revoke-keys" Oct 29 22:35:02.173204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.173192 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f45e13-d301-4a71-ba7f-d4212992222a" containerName="revoke-keys" Oct 29 22:35:02.177749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.177733 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.198901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.198874 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9"] Oct 29 22:35:02.202635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.202612 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:35:02.202812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.202666 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:35:02.279932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.279906 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.280094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.280041 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnlq\" (UniqueName: \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.280153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.280131 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.381789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.381765 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.381903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.381848 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnlq\" (UniqueName: \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.381903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.381882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.382767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.382737 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.384635 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.384605 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.407617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.407596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnlq\" (UniqueName: \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\") pod \"maas-api-revoke-sub-e8348a7a-ts5r9\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.496002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.495982 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:02.808524 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:35:02.808491 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ec18a8_ff8c_484c_930c_9ceae476480f.slice/crio-4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c WatchSource:0}: Error finding container 4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c: Status 404 returned error can't find the container with id 4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c Oct 29 22:35:02.809477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:02.809451 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9"] Oct 29 22:35:03.082293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:03.082211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" event={"ID":"10ec18a8-ff8c-484c-930c-9ceae476480f","Type":"ContainerStarted","Data":"4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c"} Oct 29 22:35:04.093938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:04.093909 4543 generic.go:356] "Generic (PLEG): container finished" podID="10ec18a8-ff8c-484c-930c-9ceae476480f" containerID="b626033d76fcb0ee4a8f275040be3c4d79702485346e9e13611467cacf256a41" exitCode=0 Oct 29 22:35:04.094316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:04.093987 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" event={"ID":"10ec18a8-ff8c-484c-930c-9ceae476480f","Type":"ContainerDied","Data":"b626033d76fcb0ee4a8f275040be3c4d79702485346e9e13611467cacf256a41"} Oct 29 22:35:05.298959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.298932 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:05.426667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.426593 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\") pod \"10ec18a8-ff8c-484c-930c-9ceae476480f\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " Oct 29 22:35:05.426846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.426809 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\" (UniqueName: \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\") pod \"10ec18a8-ff8c-484c-930c-9ceae476480f\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " Oct 29 22:35:05.426846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.426837 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\") pod \"10ec18a8-ff8c-484c-930c-9ceae476480f\" (UID: \"10ec18a8-ff8c-484c-930c-9ceae476480f\") " Oct 29 22:35:05.427009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.426983 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca" pod "10ec18a8-ff8c-484c-930c-9ceae476480f" (UID: "10ec18a8-ff8c-484c-930c-9ceae476480f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:05.427655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.427554 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:05.436987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.436931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca" pod "10ec18a8-ff8c-484c-930c-9ceae476480f" (UID: "10ec18a8-ff8c-484c-930c-9ceae476480f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:05.438065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.438034 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq" pod "10ec18a8-ff8c-484c-930c-9ceae476480f" (UID: "10ec18a8-ff8c-484c-930c-9ceae476480f"). InnerVolumeSpecName "kube-api-access-hvnlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:05.529594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.529565 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnlq\" (UniqueName: \"kubernetes.io/projected/10ec18a8-ff8c-484c-930c-9ceae476480f-kube-api-access-hvnlq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:05.529594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:05.529594 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ec18a8-ff8c-484c-930c-9ceae476480f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:06.121368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:06.121339 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" Oct 29 22:35:06.121530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:06.121373 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9" event={"ID":"10ec18a8-ff8c-484c-930c-9ceae476480f","Type":"ContainerDied","Data":"4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c"} Oct 29 22:35:06.121530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:06.121416 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b19ac5b9787fd92f7937caff5bcd9289eeb97fee1dcb9848a42616ff666166c" Oct 29 22:35:13.905979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.905939 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:13.907048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.907028 4543 cpu_manager.go:414] "containerMap: removing container" podUID="10ec18a8-ff8c-484c-930c-9ceae476480f" containerName="revoke-keys" Oct 29 22:35:13.907126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.907051 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ec18a8-ff8c-484c-930c-9ceae476480f" containerName="revoke-keys" Oct 29 22:35:13.907268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.907258 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ec18a8-ff8c-484c-930c-9ceae476480f" containerName="revoke-keys" Oct 29 22:35:13.911806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.911788 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:13.928939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.928904 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"agc-cred-904a97a988a8\"" Oct 29 22:35:13.930575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.930552 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:13.948126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.937425 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"kube-root-ca.crt\"" Oct 29 22:35:13.948126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.937458 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"openshift-service-ca.crt\"" Oct 29 22:35:13.948126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.937528 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"payload-processing-external-model-plugins-e2e-extmod-a-749f72\"" Oct 29 22:35:13.948126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:13.937616 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"routing-overlay\"" Oct 29 22:35:14.029154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.029130 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"payload-processing-external-model-e2e--d928d360-dockercfg-gp58q\"" Oct 29 22:35:14.093736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.093709 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.093906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.093891 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.094017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.093998 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.094114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.094100 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hbh\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.195844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.195777 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.195976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.195876 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.195976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.195943 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hbh\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.196184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.196071 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.196549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.196530 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.196669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.196649 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.198923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.198900 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.221366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.221342 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hbh\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.228167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.228150 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:14.550416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.550385 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:14.597199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:14.597174 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:15.259099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:15.256129 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" event={"ID":"818f16d7-1a3e-4b1f-a428-07e9476de48b","Type":"ContainerStarted","Data":"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c"} Oct 29 22:35:15.259099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:15.256232 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" event={"ID":"818f16d7-1a3e-4b1f-a428-07e9476de48b","Type":"ContainerStarted","Data":"402d3a9ed15ffe6b3eb9d4171fd4e40bc8bd5ed9d430c2585649451e3c13356d"} Oct 29 22:35:15.259099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:15.256410 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:20.256498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.256463 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" podUID="818f16d7-1a3e-4b1f-a428-07e9476de48b" containerName="payload-processing" containerID="cri-o://1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c" gracePeriod=25 Oct 29 22:35:20.620677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.620655 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:20.727274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.727246 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\") pod \"818f16d7-1a3e-4b1f-a428-07e9476de48b\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " Oct 29 22:35:20.727461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.727329 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\") pod \"818f16d7-1a3e-4b1f-a428-07e9476de48b\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " Oct 29 22:35:20.727461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.727407 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\") pod \"818f16d7-1a3e-4b1f-a428-07e9476de48b\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " Oct 29 22:35:20.727651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.727620 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\") pod \"818f16d7-1a3e-4b1f-a428-07e9476de48b\" (UID: \"818f16d7-1a3e-4b1f-a428-07e9476de48b\") " Oct 29 22:35:20.727744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.727716 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume" pod "818f16d7-1a3e-4b1f-a428-07e9476de48b" (UID: "818f16d7-1a3e-4b1f-a428-07e9476de48b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:20.728037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.728010 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume" pod "818f16d7-1a3e-4b1f-a428-07e9476de48b" (UID: "818f16d7-1a3e-4b1f-a428-07e9476de48b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:20.728421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.728329 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.728542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.728503 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/818f16d7-1a3e-4b1f-a428-07e9476de48b-routing-overlay-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.730161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.730136 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh" pod "818f16d7-1a3e-4b1f-a428-07e9476de48b" (UID: "818f16d7-1a3e-4b1f-a428-07e9476de48b"). InnerVolumeSpecName "kube-api-access-p7hbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:20.730255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.730191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials" pod "818f16d7-1a3e-4b1f-a428-07e9476de48b" (UID: "818f16d7-1a3e-4b1f-a428-07e9476de48b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:20.829864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.829812 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hbh\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-kube-api-access-p7hbh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.829864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:20.829837 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/818f16d7-1a3e-4b1f-a428-07e9476de48b-provider-credentials\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:21.316403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.316367 4543 generic.go:356] "Generic (PLEG): container finished" podID="818f16d7-1a3e-4b1f-a428-07e9476de48b" containerID="1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c" exitCode=0 Oct 29 22:35:21.316866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.316421 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" Oct 29 22:35:21.316866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.316437 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" event={"ID":"818f16d7-1a3e-4b1f-a428-07e9476de48b","Type":"ContainerDied","Data":"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c"} Oct 29 22:35:21.316866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.316497 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm" event={"ID":"818f16d7-1a3e-4b1f-a428-07e9476de48b","Type":"ContainerDied","Data":"402d3a9ed15ffe6b3eb9d4171fd4e40bc8bd5ed9d430c2585649451e3c13356d"} Oct 29 22:35:21.316866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.316524 4543 scope.go:122] "RemoveContainer" containerID="1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c" Oct 29 22:35:21.329908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.329892 4543 scope.go:122] "RemoveContainer" containerID="1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c" Oct 29 22:35:21.330199 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:35:21.330169 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c\": container with ID starting with 1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c not found: ID does not exist" containerID="1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c" Oct 29 22:35:21.330326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.330203 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c"} err="failed to get container status \"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c\": rpc error: code = NotFound desc = could not find container \"1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c\": container with ID starting with 1c1a9611c11dbe7573fd172a57b2bac176db648da67198d1f4de7e79ea6f013c not found: ID does not exist" Oct 29 22:35:21.381247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.381228 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:21.394250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:21.394231 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7ccch8bqm"] Oct 29 22:35:22.389943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:22.389912 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818f16d7-1a3e-4b1f-a428-07e9476de48b" path="/var/lib/kubelet/pods/818f16d7-1a3e-4b1f-a428-07e9476de48b/volumes" Oct 29 22:35:24.117259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:24.117228 4543 scope.go:122] "RemoveContainer" containerID="80c114b30a4c16a55df39ea035c36f6857db5956c2c40f792948243143ffe04b" Oct 29 22:35:27.962528 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:35:27.962490 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:35:45.676251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.676177 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf"] Oct 29 22:35:45.677297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.677278 4543 cpu_manager.go:414] "containerMap: removing container" podUID="818f16d7-1a3e-4b1f-a428-07e9476de48b" containerName="payload-processing" Oct 29 22:35:45.677358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.677298 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818f16d7-1a3e-4b1f-a428-07e9476de48b" containerName="payload-processing" Oct 29 22:35:45.677495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.677485 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="818f16d7-1a3e-4b1f-a428-07e9476de48b" containerName="payload-processing" Oct 29 22:35:45.683388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.683371 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.700902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.700877 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf"] Oct 29 22:35:45.707010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.706960 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:35:45.707172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.707138 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:35:45.721774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.721750 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdt24\" (UniqueName: \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.721898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.721796 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.722061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.722043 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.823340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.823314 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.823566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.823546 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.823699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.823685 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdt24\" (UniqueName: \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.824232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.824210 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.825726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.825703 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:45.849378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:45.849357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdt24\" (UniqueName: \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\") pod \"maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:46.004892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:46.004867 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:46.318271 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:35:46.318240 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe6cf16_43e9_45f3_ab66_b18953893be5.slice/crio-310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a WatchSource:0}: Error finding container 310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a: Status 404 returned error can't find the container with id 310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a Oct 29 22:35:46.318974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:46.318950 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf"] Oct 29 22:35:46.854338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:46.854306 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" event={"ID":"7fe6cf16-43e9-45f3-ab66-b18953893be5","Type":"ContainerStarted","Data":"8b134b95249576031519c70ffa0c4d46ee75a5745e5adc21c1e1bc4a0753f49c"} Oct 29 22:35:46.854717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:46.854372 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" event={"ID":"7fe6cf16-43e9-45f3-ab66-b18953893be5","Type":"ContainerStarted","Data":"310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a"} Oct 29 22:35:46.919157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:46.911069 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" podStartSLOduration=1.9110490169999998 podStartE2EDuration="1.911049017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:46.909399578 +0000 UTC m=+3208.651330251" watchObservedRunningTime="2026-10-29 22:35:46.911049017 +0000 UTC m=+3208.652979694" Oct 29 22:35:47.866319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:47.866278 4543 generic.go:356] "Generic (PLEG): container finished" podID="7fe6cf16-43e9-45f3-ab66-b18953893be5" containerID="8b134b95249576031519c70ffa0c4d46ee75a5745e5adc21c1e1bc4a0753f49c" exitCode=0 Oct 29 22:35:47.866709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:47.866420 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" event={"ID":"7fe6cf16-43e9-45f3-ab66-b18953893be5","Type":"ContainerDied","Data":"8b134b95249576031519c70ffa0c4d46ee75a5745e5adc21c1e1bc4a0753f49c"} Oct 29 22:35:49.058363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.058336 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:49.197923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.197848 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\") pod \"7fe6cf16-43e9-45f3-ab66-b18953893be5\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " Oct 29 22:35:49.198207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.198185 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\") pod \"7fe6cf16-43e9-45f3-ab66-b18953893be5\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " Oct 29 22:35:49.198257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.198235 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca" pod "7fe6cf16-43e9-45f3-ab66-b18953893be5" (UID: "7fe6cf16-43e9-45f3-ab66-b18953893be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:49.198309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.198294 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\" (UniqueName: \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\") pod \"7fe6cf16-43e9-45f3-ab66-b18953893be5\" (UID: \"7fe6cf16-43e9-45f3-ab66-b18953893be5\") " Oct 29 22:35:49.198988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.198966 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:49.200452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.200427 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24" pod "7fe6cf16-43e9-45f3-ab66-b18953893be5" (UID: "7fe6cf16-43e9-45f3-ab66-b18953893be5"). InnerVolumeSpecName "kube-api-access-jdt24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:49.200545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.200481 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca" pod "7fe6cf16-43e9-45f3-ab66-b18953893be5" (UID: "7fe6cf16-43e9-45f3-ab66-b18953893be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:49.300651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.300628 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe6cf16-43e9-45f3-ab66-b18953893be5-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:49.300651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.300652 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdt24\" (UniqueName: \"kubernetes.io/projected/7fe6cf16-43e9-45f3-ab66-b18953893be5-kube-api-access-jdt24\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:35:49.886660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.886626 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" Oct 29 22:35:49.886660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.886646 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf" event={"ID":"7fe6cf16-43e9-45f3-ab66-b18953893be5","Type":"ContainerDied","Data":"310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a"} Oct 29 22:35:49.886876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:49.886691 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310415a1bc98cfa397222df58770d884c3a3eb289908f8f68bebcf2fdce1c24a" Oct 29 22:35:56.156865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:56.156832 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:35:56.194586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:35:56.194552 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:35:57.962915 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:35:57.962882 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:36:01.157637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.157598 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" podUID="f608efa0-2cf1-4c5c-b15f-7230fddac846" containerName="payload-processing" containerID="cri-o://ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c" gracePeriod=25 Oct 29 22:36:01.195155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.195121 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" podUID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" containerName="payload-pre-processing" containerID="cri-o://96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037" gracePeriod=25 Oct 29 22:36:01.541816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.541793 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:36:01.546004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.545985 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:36:01.606779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.606744 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:36:01.607841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607820 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f608efa0-2cf1-4c5c-b15f-7230fddac846" containerName="payload-processing" Oct 29 22:36:01.607841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607842 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f608efa0-2cf1-4c5c-b15f-7230fddac846" containerName="payload-processing" Oct 29 22:36:01.608024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607861 4543 cpu_manager.go:414] "containerMap: removing container" podUID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" containerName="payload-pre-processing" Oct 29 22:36:01.608024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607868 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" containerName="payload-pre-processing" Oct 29 22:36:01.608024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607957 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7fe6cf16-43e9-45f3-ab66-b18953893be5" containerName="revoke-keys" Oct 29 22:36:01.608024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.607964 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe6cf16-43e9-45f3-ab66-b18953893be5" containerName="revoke-keys" Oct 29 22:36:01.608268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.608255 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe6cf16-43e9-45f3-ab66-b18953893be5" containerName="revoke-keys" Oct 29 22:36:01.608371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.608282 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f608efa0-2cf1-4c5c-b15f-7230fddac846" containerName="payload-processing" Oct 29 22:36:01.608371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.608293 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" containerName="payload-pre-processing" Oct 29 22:36:01.612590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.612572 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.633611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.633576 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-26abe8-openshift-default-dockercfg-frqz6\"" Oct 29 22:36:01.633728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.633678 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\") pod \"f608efa0-2cf1-4c5c-b15f-7230fddac846\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " Oct 29 22:36:01.634013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.633993 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume" pod "f608efa0-2cf1-4c5c-b15f-7230fddac846" (UID: "f608efa0-2cf1-4c5c-b15f-7230fddac846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:01.634233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.634211 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\" (UniqueName: \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\") pod \"f608efa0-2cf1-4c5c-b15f-7230fddac846\" (UID: \"f608efa0-2cf1-4c5c-b15f-7230fddac846\") " Oct 29 22:36:01.634353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.634276 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\") pod \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " Oct 29 22:36:01.634353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.634320 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\" (UniqueName: \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\") pod \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\" (UID: \"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7\") " Oct 29 22:36:01.634914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.634889 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume" pod "266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" (UID: "266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:01.635239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.635216 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f608efa0-2cf1-4c5c-b15f-7230fddac846-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:01.635239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.635245 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:01.639010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.637433 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4" pod "f608efa0-2cf1-4c5c-b15f-7230fddac846" (UID: "f608efa0-2cf1-4c5c-b15f-7230fddac846"). InnerVolumeSpecName "kube-api-access-pd8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:01.639010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.637480 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:36:01.639010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.638852 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6" pod "266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" (UID: "266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7"). InnerVolumeSpecName "kube-api-access-c4pd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:01.737019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.736932 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737041 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737203 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737230 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737339 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7k9\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737429 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737631 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.737865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737852 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8r4\" (UniqueName: \"kubernetes.io/projected/f608efa0-2cf1-4c5c-b15f-7230fddac846-kube-api-access-pd8r4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:01.737900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.737869 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4pd6\" (UniqueName: \"kubernetes.io/projected/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7-kube-api-access-c4pd6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:01.839025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.838994 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839135 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7k9\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839193 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839392 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839422 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839451 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839557 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839732 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839781 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.839878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839840 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.840013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.839993 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.840600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.840577 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.841686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.841667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.841798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.841783 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.864662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.864642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.864828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.864811 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7k9\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:01.936061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:01.936043 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:02.059815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.059778 4543 generic.go:356] "Generic (PLEG): container finished" podID="f608efa0-2cf1-4c5c-b15f-7230fddac846" containerID="ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c" exitCode=0 Oct 29 22:36:02.059977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.059826 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" Oct 29 22:36:02.059977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.059868 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" event={"ID":"f608efa0-2cf1-4c5c-b15f-7230fddac846","Type":"ContainerDied","Data":"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c"} Oct 29 22:36:02.059977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.059925 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd" event={"ID":"f608efa0-2cf1-4c5c-b15f-7230fddac846","Type":"ContainerDied","Data":"3b384e028826ef785035231071bc8e032da7c55ccc7c392fb362360385f7eaa0"} Oct 29 22:36:02.059977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.059956 4543 scope.go:122] "RemoveContainer" containerID="ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c" Oct 29 22:36:02.062676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.062648 4543 generic.go:356] "Generic (PLEG): container finished" podID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" containerID="96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037" exitCode=0 Oct 29 22:36:02.062808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.062692 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" Oct 29 22:36:02.062808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.062740 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" event={"ID":"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7","Type":"ContainerDied","Data":"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037"} Oct 29 22:36:02.062808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.062792 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2" event={"ID":"266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7","Type":"ContainerDied","Data":"97888b3122fe0a082e5fd1d25217b30646f7d8a98fde552c4e7f67a98c2391a7"} Oct 29 22:36:02.075973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.075865 4543 scope.go:122] "RemoveContainer" containerID="ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c" Oct 29 22:36:02.076319 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:36:02.076295 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c\": container with ID starting with ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c not found: ID does not exist" containerID="ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c" Oct 29 22:36:02.076550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.076327 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c"} err="failed to get container status \"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c\": rpc error: code = NotFound desc = could not find container \"ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c\": container with ID starting with ee43921bc5d24a992ae6c29a957ed94b7839c66ca5a9158baa5eb9df5112ef3c not found: ID does not exist" Oct 29 22:36:02.076550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.076351 4543 scope.go:122] "RemoveContainer" containerID="96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037" Oct 29 22:36:02.093603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.093510 4543 scope.go:122] "RemoveContainer" containerID="96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037" Oct 29 22:36:02.093989 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:36:02.093949 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037\": container with ID starting with 96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037 not found: ID does not exist" containerID="96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037" Oct 29 22:36:02.094179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.093983 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037"} err="failed to get container status \"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037\": rpc error: code = NotFound desc = could not find container \"96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037\": container with ID starting with 96ef112aecbf42107456f6bb337593c5a305450b7c230417ec0468db152c4037 not found: ID does not exist" Oct 29 22:36:02.129124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.128627 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:36:02.138898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.138867 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cb13f-7d59c466f8-dxpmd"] Oct 29 22:36:02.170965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.170935 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:36:02.176545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.176523 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cb13f-567f87d6cf-mnmn2"] Oct 29 22:36:02.281673 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:36:02.281649 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0742b7e5_5e61_46b7_8c25_18cd3bee9d69.slice/crio-2f11492dd3c04d79713f37d7c944039e35fdb51fcfa84c29b986f51b4e231049 WatchSource:0}: Error finding container 2f11492dd3c04d79713f37d7c944039e35fdb51fcfa84c29b986f51b4e231049: Status 404 returned error can't find the container with id 2f11492dd3c04d79713f37d7c944039e35fdb51fcfa84c29b986f51b4e231049 Oct 29 22:36:02.282032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.282011 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:36:02.285287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.285255 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:36:02.285380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.285318 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:36:02.285380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.285350 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:36:02.390951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.390927 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7" path="/var/lib/kubelet/pods/266ee6eb-04ae-4db4-9bb1-1e0e4d6406c7/volumes" Oct 29 22:36:02.391465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:02.391447 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f608efa0-2cf1-4c5c-b15f-7230fddac846" path="/var/lib/kubelet/pods/f608efa0-2cf1-4c5c-b15f-7230fddac846/volumes" Oct 29 22:36:03.075143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.075099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" event={"ID":"0742b7e5-5e61-46b7-8c25-18cd3bee9d69","Type":"ContainerStarted","Data":"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e"} Oct 29 22:36:03.075340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.075157 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" event={"ID":"0742b7e5-5e61-46b7-8c25-18cd3bee9d69","Type":"ContainerStarted","Data":"2f11492dd3c04d79713f37d7c944039e35fdb51fcfa84c29b986f51b4e231049"} Oct 29 22:36:03.154155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.151985 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podStartSLOduration=49.151960739 podStartE2EDuration="49.151960739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:03.123983099 +0000 UTC m=+3224.865913771" watchObservedRunningTime="2026-10-29 22:36:03.151960739 +0000 UTC m=+3224.893891414" Oct 29 22:36:03.953169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.953130 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:03.953641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.953210 4543 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" start-of-body= Oct 29 22:36:03.953641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:03.953265 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" Oct 29 22:36:04.945101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:04.941390 4543 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" start-of-body= Oct 29 22:36:04.945101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:04.941450 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" Oct 29 22:36:05.953226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:05.953184 4543 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" start-of-body= Oct 29 22:36:05.953658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:05.953236 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" Oct 29 22:36:06.953212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:06.950168 4543 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" start-of-body= Oct 29 22:36:06.953212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:06.950234 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" Oct 29 22:36:07.954259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:07.954190 4543 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" start-of-body= Oct 29 22:36:07.954756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:07.954263 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.159:15021/healthz/ready\": dial tcp 10.133.1.159:15021: connect: connection refused" Oct 29 22:36:08.943321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:08.943283 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:08.943615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:08.943595 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:08.944261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:08.944242 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:36:10.038604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:10.038570 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv"] Oct 29 22:36:10.051698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:10.051663 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8badc926-5jwlv"] Oct 29 22:36:10.389328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:10.389260 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb97bda-7946-4104-93ea-db0a353bea3b" path="/var/lib/kubelet/pods/bcb97bda-7946-4104-93ea-db0a353bea3b/volumes" Oct 29 22:36:21.046331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:21.046289 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644"] Oct 29 22:36:21.057176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:21.057140 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8b7824-78644"] Oct 29 22:36:22.389255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:22.389225 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff603f0-e6d0-4568-b64d-f8e6428e4ff8" path="/var/lib/kubelet/pods/4ff603f0-e6d0-4568-b64d-f8e6428e4ff8/volumes" Oct 29 22:36:26.362128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:26.362094 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf"] Oct 29 22:36:26.374931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:26.374905 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cb13f-8vqnf"] Oct 29 22:36:26.388639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:26.388611 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe6cf16-43e9-45f3-ab66-b18953893be5" path="/var/lib/kubelet/pods/7fe6cf16-43e9-45f3-ab66-b18953893be5/volumes" Oct 29 22:36:27.963339 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:36:27.963299 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:36:36.094535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.094504 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z"] Oct 29 22:36:36.101658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.101634 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.119378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.119357 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z"] Oct 29 22:36:36.124784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.124759 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:36:36.124898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.124813 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:36:36.261470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.261442 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlls4\" (UniqueName: \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.261617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.261562 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.261675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.261631 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.363325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.363261 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.363444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.363345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.363498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.363487 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlls4\" (UniqueName: \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.364138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.364062 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.366037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.366011 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.389482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.389452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlls4\" (UniqueName: \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\") pod \"maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.421217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.421198 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:36.736333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:36.736308 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z"] Oct 29 22:36:36.736810 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:36:36.736786 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaa7fab_b43f_4741_9d8e_e31cbfcfd06f.slice/crio-e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332 WatchSource:0}: Error finding container e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332: Status 404 returned error can't find the container with id e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332 Oct 29 22:36:37.619137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:37.619062 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" event={"ID":"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f","Type":"ContainerStarted","Data":"a870ec98b2249939f2acab1a954d99289e0e4585c7bdcf4221c9be4e1fd1d0e1"} Oct 29 22:36:37.619503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:37.619169 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" event={"ID":"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f","Type":"ContainerStarted","Data":"e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332"} Oct 29 22:36:37.665655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:37.665614 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" podStartSLOduration=1.6656013729999999 podStartE2EDuration="1.665601373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:37.662976409 +0000 UTC m=+3259.404907083" watchObservedRunningTime="2026-10-29 22:36:37.665601373 +0000 UTC m=+3259.407532045" Oct 29 22:36:38.629565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:38.629535 4543 generic.go:356] "Generic (PLEG): container finished" podID="dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" containerID="a870ec98b2249939f2acab1a954d99289e0e4585c7bdcf4221c9be4e1fd1d0e1" exitCode=0 Oct 29 22:36:38.629979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:38.629616 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" event={"ID":"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f","Type":"ContainerDied","Data":"a870ec98b2249939f2acab1a954d99289e0e4585c7bdcf4221c9be4e1fd1d0e1"} Oct 29 22:36:39.834598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.834570 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:39.918377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.918298 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\") pod \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " Oct 29 22:36:39.918377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.918360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\" (UniqueName: \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\") pod \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " Oct 29 22:36:39.918629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.918557 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\") pod \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\" (UID: \"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f\") " Oct 29 22:36:39.919008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.918935 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca" pod "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" (UID: "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:39.919301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.919281 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:39.921321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.921297 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca" pod "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" (UID: "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:39.921427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:39.921372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4" pod "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" (UID: "dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f"). InnerVolumeSpecName "kube-api-access-zlls4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:40.020516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:40.020492 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:40.020516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:40.020516 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlls4\" (UniqueName: \"kubernetes.io/projected/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f-kube-api-access-zlls4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:40.649893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:40.649826 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" Oct 29 22:36:40.649893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:40.649861 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z" event={"ID":"dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f","Type":"ContainerDied","Data":"e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332"} Oct 29 22:36:40.650069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:40.649912 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99b4e945f606aad715d817fafd1a3615ebd96a2cb900119415336c91792a332" Oct 29 22:36:46.589226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:46.589189 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:36:46.629913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:46.629887 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:36:50.039362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:50.039324 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j"] Oct 29 22:36:50.045313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:50.045286 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fe183b-4ns8j"] Oct 29 22:36:50.402048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:50.401966 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d91908-86e6-4da8-8c2b-3841df3ca4c7" path="/var/lib/kubelet/pods/84d91908-86e6-4da8-8c2b-3841df3ca4c7/volumes" Oct 29 22:36:51.040127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.040088 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km"] Oct 29 22:36:51.053659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.053624 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6752a6-zm2km"] Oct 29 22:36:51.590389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.590351 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" containerID="cri-o://2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af" gracePeriod=25 Oct 29 22:36:51.628734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.628703 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln"] Oct 29 22:36:51.630342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.630318 4543 cpu_manager.go:414] "containerMap: removing container" podUID="dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" containerName="revoke-keys" Oct 29 22:36:51.630522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.630498 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" containerName="revoke-keys" Oct 29 22:36:51.631010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.630990 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" containerName="revoke-keys" Oct 29 22:36:51.631200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.630359 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" containerID="cri-o://e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95" gracePeriod=25 Oct 29 22:36:51.636883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.636860 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.654131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.654058 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln"] Oct 29 22:36:51.659338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.659319 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:36:51.659472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.659358 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:36:51.694464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.694430 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d57t\" (UniqueName: \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.694599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.694538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.694667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.694651 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.796134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.796063 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d57t\" (UniqueName: \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.796352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.796155 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.796352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.796241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.796846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.796820 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.798837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.798809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.822321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.822294 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d57t\" (UniqueName: \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\") pod \"maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:51.962069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:51.962049 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:52.286649 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:36:52.286620 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc72caf_1d1d_40e4_a261_c242b0287762.slice/crio-d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6 WatchSource:0}: Error finding container d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6: Status 404 returned error can't find the container with id d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6 Oct 29 22:36:52.287067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:52.286866 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln"] Oct 29 22:36:52.389661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:52.389631 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792" path="/var/lib/kubelet/pods/00bf9ceb-e7a4-4ac5-bf8a-ea9c088d7792/volumes" Oct 29 22:36:52.788653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:52.788622 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" event={"ID":"5fc72caf-1d1d-40e4-a261-c242b0287762","Type":"ContainerStarted","Data":"bd2b412f55b05a669abef97447fe0e288b0570b59a9c9ecde72009b0832bdb5e"} Oct 29 22:36:52.788771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:52.788682 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" event={"ID":"5fc72caf-1d1d-40e4-a261-c242b0287762","Type":"ContainerStarted","Data":"d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6"} Oct 29 22:36:52.835153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:52.835068 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" podStartSLOduration=1.835053813 podStartE2EDuration="1.835053813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:52.831804348 +0000 UTC m=+3274.573735022" watchObservedRunningTime="2026-10-29 22:36:52.835053813 +0000 UTC m=+3274.576984485" Oct 29 22:36:53.803714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:53.803020 4543 generic.go:356] "Generic (PLEG): container finished" podID="5fc72caf-1d1d-40e4-a261-c242b0287762" containerID="bd2b412f55b05a669abef97447fe0e288b0570b59a9c9ecde72009b0832bdb5e" exitCode=0 Oct 29 22:36:53.803714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:53.803273 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" event={"ID":"5fc72caf-1d1d-40e4-a261-c242b0287762","Type":"ContainerDied","Data":"bd2b412f55b05a669abef97447fe0e288b0570b59a9c9ecde72009b0832bdb5e"} Oct 29 22:36:54.999329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:54.999302 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:55.061377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.061341 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\") pod \"5fc72caf-1d1d-40e4-a261-c242b0287762\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " Oct 29 22:36:55.061534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.061474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\" (UniqueName: \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\") pod \"5fc72caf-1d1d-40e4-a261-c242b0287762\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " Oct 29 22:36:55.061609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.061544 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\") pod \"5fc72caf-1d1d-40e4-a261-c242b0287762\" (UID: \"5fc72caf-1d1d-40e4-a261-c242b0287762\") " Oct 29 22:36:55.062460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.062431 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca" pod "5fc72caf-1d1d-40e4-a261-c242b0287762" (UID: "5fc72caf-1d1d-40e4-a261-c242b0287762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:55.062665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.062635 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:55.064553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.064520 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca" pod "5fc72caf-1d1d-40e4-a261-c242b0287762" (UID: "5fc72caf-1d1d-40e4-a261-c242b0287762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:55.064701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.064674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t" pod "5fc72caf-1d1d-40e4-a261-c242b0287762" (UID: "5fc72caf-1d1d-40e4-a261-c242b0287762"). InnerVolumeSpecName "kube-api-access-7d57t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:55.163805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.163769 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d57t\" (UniqueName: \"kubernetes.io/projected/5fc72caf-1d1d-40e4-a261-c242b0287762-kube-api-access-7d57t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:55.163805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.163803 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc72caf-1d1d-40e4-a261-c242b0287762-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:36:55.828872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.828840 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" Oct 29 22:36:55.828872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.828854 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln" event={"ID":"5fc72caf-1d1d-40e4-a261-c242b0287762","Type":"ContainerDied","Data":"d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6"} Oct 29 22:36:55.829106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:55.828906 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d086df6031a34f922c4ef042ed731c5937b90a4336673043bcf860e1498664c6" Oct 29 22:36:57.964723 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:36:57.964682 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:36:59.451131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:59.451040 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:36:59.451570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:36:59.451052 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:00.041220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:00.041186 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq"] Oct 29 22:37:00.053925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:00.053893 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90416f56-4zqfq"] Oct 29 22:37:00.390466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:00.390394 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2b9bf2-b313-4f1c-8e40-570049284e49" path="/var/lib/kubelet/pods/ed2b9bf2-b313-4f1c-8e40-570049284e49/volumes" Oct 29 22:37:01.036151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:01.036121 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc"] Oct 29 22:37:01.053016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:01.052987 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a790551-2rttc"] Oct 29 22:37:02.094331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:02.094302 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:37:02.431838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:02.431753 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a325ac-8170-44a7-a768-a4ca86e5890d" path="/var/lib/kubelet/pods/59a325ac-8170-44a7-a768-a4ca86e5890d/volumes" Oct 29 22:37:07.095213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.095178 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" podUID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" containerName="payload-processing" containerID="cri-o://9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42" gracePeriod=25 Oct 29 22:37:07.397808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.397784 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:37:07.483139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.483110 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:37:07.485856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.485827 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" containerName="payload-processing" Oct 29 22:37:07.486102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.486064 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" containerName="payload-processing" Oct 29 22:37:07.486243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.486228 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5fc72caf-1d1d-40e4-a261-c242b0287762" containerName="revoke-keys" Oct 29 22:37:07.486346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.486333 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc72caf-1d1d-40e4-a261-c242b0287762" containerName="revoke-keys" Oct 29 22:37:07.487046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.487024 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" containerName="payload-processing" Oct 29 22:37:07.487213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.487197 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc72caf-1d1d-40e4-a261-c242b0287762" containerName="revoke-keys" Oct 29 22:37:07.494604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.494583 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.512550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.512522 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-27b6db\"" Oct 29 22:37:07.512670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.512634 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-27b6db-dockercfg-g7zwb\"" Oct 29 22:37:07.517982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.517958 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:37:07.518292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.518271 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\" (UniqueName: \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\") pod \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " Oct 29 22:37:07.518410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.518393 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\") pod \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\" (UID: \"5ac3f9e6-12af-4993-8a74-e57d3a428b85\") " Oct 29 22:37:07.518796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.518772 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume" pod "5ac3f9e6-12af-4993-8a74-e57d3a428b85" (UID: "5ac3f9e6-12af-4993-8a74-e57d3a428b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:07.521027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.520989 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq" pod "5ac3f9e6-12af-4993-8a74-e57d3a428b85" (UID: "5ac3f9e6-12af-4993-8a74-e57d3a428b85"). InnerVolumeSpecName "kube-api-access-nrrdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:07.522035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.522007 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrdq\" (UniqueName: \"kubernetes.io/projected/5ac3f9e6-12af-4993-8a74-e57d3a428b85-kube-api-access-nrrdq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:07.522171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.522039 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac3f9e6-12af-4993-8a74-e57d3a428b85-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:07.623654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.623588 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngxd\" (UniqueName: \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.623870 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.623849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.725427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.725390 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngxd\" (UniqueName: \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.725812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.725792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.726510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.726491 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.751180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.751157 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngxd\" (UniqueName: \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\") pod \"payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:07.814013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:07.813993 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:08.088393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.088352 4543 generic.go:356] "Generic (PLEG): container finished" podID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" containerID="9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42" exitCode=0 Oct 29 22:37:08.088650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.088626 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" event={"ID":"5ac3f9e6-12af-4993-8a74-e57d3a428b85","Type":"ContainerDied","Data":"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42"} Oct 29 22:37:08.088750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.088685 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" event={"ID":"5ac3f9e6-12af-4993-8a74-e57d3a428b85","Type":"ContainerDied","Data":"0a339a80f911f2c64acf8082cf80b0c326b5c4c175c9c1775154a0b53417021e"} Oct 29 22:37:08.088750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.088717 4543 scope.go:122] "RemoveContainer" containerID="9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42" Oct 29 22:37:08.088948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.088930 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng" Oct 29 22:37:08.108547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.108296 4543 scope.go:122] "RemoveContainer" containerID="9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42" Oct 29 22:37:08.108859 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:08.108601 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42\": container with ID starting with 9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42 not found: ID does not exist" containerID="9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42" Oct 29 22:37:08.108859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.108631 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42"} err="failed to get container status \"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42\": rpc error: code = NotFound desc = could not find container \"9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42\": container with ID starting with 9aa40416b2779ffa08bf60807762af75d9a9ed2f0c57453f68175e12227bdf42 not found: ID does not exist" Oct 29 22:37:08.139492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.139466 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:37:08.139932 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:08.139910 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61ee2c6_6331_43ce_9ac3_942625eda128.slice/crio-d7672ddad5b1ef4c9b00992933559983ff310dd33f952f51fee0069c4428ce0d WatchSource:0}: Error finding container d7672ddad5b1ef4c9b00992933559983ff310dd33f952f51fee0069c4428ce0d: Status 404 returned error can't find the container with id d7672ddad5b1ef4c9b00992933559983ff310dd33f952f51fee0069c4428ce0d Oct 29 22:37:08.177520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.177148 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:37:08.185531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.185508 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-219ccc-659b44f4c8-jspng"] Oct 29 22:37:08.389924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:08.389851 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac3f9e6-12af-4993-8a74-e57d3a428b85" path="/var/lib/kubelet/pods/5ac3f9e6-12af-4993-8a74-e57d3a428b85/volumes" Oct 29 22:37:09.103516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.103481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" event={"ID":"d61ee2c6-6331-43ce-9ac3-942625eda128","Type":"ContainerStarted","Data":"0cfb1ec39ade7c110fde493ca4a2c4790e10fe0308635b7d13261d0643f8565d"} Oct 29 22:37:09.103719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.103540 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:09.103719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.103562 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" event={"ID":"d61ee2c6-6331-43ce-9ac3-942625eda128","Type":"ContainerStarted","Data":"d7672ddad5b1ef4c9b00992933559983ff310dd33f952f51fee0069c4428ce0d"} Oct 29 22:37:09.160354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.160289 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" podStartSLOduration=38.160275476 podStartE2EDuration="38.160275476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:09.154912604 +0000 UTC m=+3290.896843278" watchObservedRunningTime="2026-10-29 22:37:09.160275476 +0000 UTC m=+3290.902206148" Oct 29 22:37:09.450708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.450628 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:09.450861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:09.450640 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:11.953586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.953554 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:37:11.970593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.970564 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\" (UniqueName: \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\") pod \"63633283-238a-4630-975a-b9bb3c43d9b9\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " Oct 29 22:37:11.970725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.970709 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\") pod \"63633283-238a-4630-975a-b9bb3c43d9b9\" (UID: \"63633283-238a-4630-975a-b9bb3c43d9b9\") " Oct 29 22:37:11.971067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.971036 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume" pod "63633283-238a-4630-975a-b9bb3c43d9b9" (UID: "63633283-238a-4630-975a-b9bb3c43d9b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:11.971844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.971812 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63633283-238a-4630-975a-b9bb3c43d9b9-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:11.974046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:11.974019 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n" pod "63633283-238a-4630-975a-b9bb3c43d9b9" (UID: "63633283-238a-4630-975a-b9bb3c43d9b9"). InnerVolumeSpecName "kube-api-access-c4v9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:12.056782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.056750 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:37:12.073528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.073500 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\") pod \"18e8126b-2602-4a76-af63-860ae670c6d3\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " Oct 29 22:37:12.073677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.073635 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\" (UniqueName: \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\") pod \"18e8126b-2602-4a76-af63-860ae670c6d3\" (UID: \"18e8126b-2602-4a76-af63-860ae670c6d3\") " Oct 29 22:37:12.073961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.073933 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume" pod "18e8126b-2602-4a76-af63-860ae670c6d3" (UID: "18e8126b-2602-4a76-af63-860ae670c6d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:12.074611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.074588 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4v9n\" (UniqueName: \"kubernetes.io/projected/63633283-238a-4630-975a-b9bb3c43d9b9-kube-api-access-c4v9n\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.074681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.074619 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e8126b-2602-4a76-af63-860ae670c6d3-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.076128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.076106 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn" pod "18e8126b-2602-4a76-af63-860ae670c6d3" (UID: "18e8126b-2602-4a76-af63-860ae670c6d3"). InnerVolumeSpecName "kube-api-access-x2vsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:12.118038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.117982 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:37:12.119022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119004 4543 cpu_manager.go:414] "containerMap: removing container" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" Oct 29 22:37:12.119069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119026 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" Oct 29 22:37:12.119136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119093 4543 cpu_manager.go:414] "containerMap: removing container" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" Oct 29 22:37:12.119136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119103 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" Oct 29 22:37:12.119441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119427 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" containerName="payload-pre-processing" Oct 29 22:37:12.119479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.119458 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" containerName="payload-processing" Oct 29 22:37:12.123993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.123973 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.144388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.143415 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-563f39-openshift-default-dockercfg-jvdzl\"" Oct 29 22:37:12.151445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.146854 4543 generic.go:356] "Generic (PLEG): container finished" podID="63633283-238a-4630-975a-b9bb3c43d9b9" containerID="2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af" exitCode=0 Oct 29 22:37:12.151445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.147057 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" Oct 29 22:37:12.156109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.153425 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" event={"ID":"63633283-238a-4630-975a-b9bb3c43d9b9","Type":"ContainerDied","Data":"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af"} Oct 29 22:37:12.156109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.153485 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r" event={"ID":"63633283-238a-4630-975a-b9bb3c43d9b9","Type":"ContainerDied","Data":"ef12e84ba343cd18acc773569beffa06438aad3a268161eee24b57a0ff540ac7"} Oct 29 22:37:12.156109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.153515 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:37:12.156109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.153552 4543 scope.go:122] "RemoveContainer" containerID="2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af" Oct 29 22:37:12.156109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.156101 4543 generic.go:356] "Generic (PLEG): container finished" podID="18e8126b-2602-4a76-af63-860ae670c6d3" containerID="e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95" exitCode=0 Oct 29 22:37:12.156534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.156443 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" event={"ID":"18e8126b-2602-4a76-af63-860ae670c6d3","Type":"ContainerDied","Data":"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95"} Oct 29 22:37:12.156534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.156500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" event={"ID":"18e8126b-2602-4a76-af63-860ae670c6d3","Type":"ContainerDied","Data":"1d4fd7711a3d6f05a0664e1d42c217b9aebd13d0a477b4a16042eb10c4750605"} Oct 29 22:37:12.156709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.156590 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw" Oct 29 22:37:12.175287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.174947 4543 scope.go:122] "RemoveContainer" containerID="2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af" Oct 29 22:37:12.175404 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:12.175361 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af\": container with ID starting with 2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af not found: ID does not exist" containerID="2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af" Oct 29 22:37:12.175475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.175395 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af"} err="failed to get container status \"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af\": rpc error: code = NotFound desc = could not find container \"2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af\": container with ID starting with 2500c486b7ae5d14ce2f5e5c08f040a5bdc7ecda4f2d80c7dc3afb91e246d5af not found: ID does not exist" Oct 29 22:37:12.175475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.175418 4543 scope.go:122] "RemoveContainer" containerID="e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95" Oct 29 22:37:12.176506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176179 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176344 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176427 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nkw\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176555 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176662 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176702 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176823 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.176939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176881 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.177157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.176971 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vsn\" (UniqueName: \"kubernetes.io/projected/18e8126b-2602-4a76-af63-860ae670c6d3-kube-api-access-x2vsn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.220862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.210741 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z"] Oct 29 22:37:12.220862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.219892 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b9ed94-2cr4z"] Oct 29 22:37:12.250839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.250799 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:37:12.254750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.254729 4543 scope.go:122] "RemoveContainer" containerID="e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95" Oct 29 22:37:12.255156 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:12.255129 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95\": container with ID starting with e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95 not found: ID does not exist" containerID="e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95" Oct 29 22:37:12.255253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.255167 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95"} err="failed to get container status \"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95\": rpc error: code = NotFound desc = could not find container \"e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95\": container with ID starting with e019e9579eaf51354dc4d2082571c937c04c2f96312dba3dd5c32a30346e7c95 not found: ID does not exist" Oct 29 22:37:12.257563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.257543 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b9ed94-5bc495f774-2lm7r"] Oct 29 22:37:12.272794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.272769 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:37:12.278398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278371 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nkw\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278493 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278594 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278661 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278866 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.278962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278950 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.278957 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279052 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279129 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279180 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279180 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.279718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.279661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.280272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.280253 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b9ed94-5bb8f74755-fq9gw"] Oct 29 22:37:12.281398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.281370 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.281502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.281457 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.304430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.304408 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.304519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.304437 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nkw\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\") pod \"e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.417805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.417716 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e8126b-2602-4a76-af63-860ae670c6d3" path="/var/lib/kubelet/pods/18e8126b-2602-4a76-af63-860ae670c6d3/volumes" Oct 29 22:37:12.418385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.418363 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63633283-238a-4630-975a-b9bb3c43d9b9" path="/var/lib/kubelet/pods/63633283-238a-4630-975a-b9bb3c43d9b9/volumes" Oct 29 22:37:12.419011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.418993 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f" path="/var/lib/kubelet/pods/dbaa7fab-b43f-4741-9d8e-e31cbfcfd06f/volumes" Oct 29 22:37:12.447718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.447696 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:12.781701 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:12.781675 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bbf528_f2f4_4ca7_8e9f_f09af5d74303.slice/crio-b1fb834f4d2bc2df903f72b564830206191593b62abf7ef37ad5e73e9a7c8647 WatchSource:0}: Error finding container b1fb834f4d2bc2df903f72b564830206191593b62abf7ef37ad5e73e9a7c8647: Status 404 returned error can't find the container with id b1fb834f4d2bc2df903f72b564830206191593b62abf7ef37ad5e73e9a7c8647 Oct 29 22:37:12.782884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.782855 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:37:12.785020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.784988 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:12.785120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.785055 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:12.785162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:12.785120 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:13.169645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:13.169600 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerStarted","Data":"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83"} Oct 29 22:37:13.170199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:13.169667 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerStarted","Data":"b1fb834f4d2bc2df903f72b564830206191593b62abf7ef37ad5e73e9a7c8647"} Oct 29 22:37:13.245479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:13.245434 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podStartSLOduration=170.245419402 podStartE2EDuration="2m50.245419402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:13.23184584 +0000 UTC m=+3294.973776525" watchObservedRunningTime="2026-10-29 22:37:13.245419402 +0000 UTC m=+3294.987350075" Oct 29 22:37:13.448302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:13.448224 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:13.455004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:13.454979 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:14.208101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:14.206833 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:14.210500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:14.210271 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:37:15.038720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:15.038686 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r"] Oct 29 22:37:15.051059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:15.051031 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f43767f-r8c8r"] Oct 29 22:37:16.389533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:16.389498 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c632767-2bb2-4264-b04f-39661ce36edc" path="/var/lib/kubelet/pods/1c632767-2bb2-4264-b04f-39661ce36edc/volumes" Oct 29 22:37:17.040319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:17.040283 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb"] Oct 29 22:37:17.053275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:17.053248 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b668fb5-7vztb"] Oct 29 22:37:18.389505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:18.389475 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4" path="/var/lib/kubelet/pods/8c4a6fab-33ff-4d33-80ba-8e37f4a95cd4/volumes" Oct 29 22:37:20.116341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:20.116309 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:37:20.288149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:20.284038 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:37:20.472176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:20.461818 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" containerID="cri-o://3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500" gracePeriod=30 Oct 29 22:37:24.186470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:24.186441 4543 scope.go:122] "RemoveContainer" containerID="2ca5b95f9ead51a2fb5cd23856bbb79b7fcf82af37bf40d4e167308e37a0fd39" Oct 29 22:37:24.218973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:24.218484 4543 scope.go:122] "RemoveContainer" containerID="241bc851dd14bbed2eada9e71ef64b26ac68bbb3dec3f6ca30b1a3fb365fbaf2" Oct 29 22:37:25.039691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:25.039656 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s"] Oct 29 22:37:25.053371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:25.053342 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v"] Oct 29 22:37:25.058151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:25.058129 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b4f2bf-km26s"] Oct 29 22:37:25.062617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:25.062594 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2c35b1-j9v8v"] Oct 29 22:37:26.234232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.234202 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:37:26.393791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.393703 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.395402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.395378 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.395987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.395966 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.396465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.396447 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.402360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.401823 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.412216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.412191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:26.419679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.419267 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:37:26.419679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.419398 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "kube-api-access-sfjcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:26.424296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.424059 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f906c9-bc06-4019-8fa9-0ae5d3ea2890" path="/var/lib/kubelet/pods/40f906c9-bc06-4019-8fa9-0ae5d3ea2890/volumes" Oct 29 22:37:26.424867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.424842 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1" path="/var/lib/kubelet/pods/cf9f1461-9a1e-4d94-ae46-f7c7d420f0d1/volumes" Oct 29 22:37:26.450793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.450770 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:37:26.452549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.452518 4543 cpu_manager.go:414] "containerMap: removing container" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" Oct 29 22:37:26.452648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.452550 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" Oct 29 22:37:26.452893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.452874 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerName="istio-proxy" Oct 29 22:37:26.459247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.459211 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:37:26.459357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.459304 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.481986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.481968 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-563f39-dockercfg-kkt9d\"" Oct 29 22:37:26.482115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.482007 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-563f39\"" Oct 29 22:37:26.487030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.487010 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx"] Oct 29 22:37:26.492925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.492873 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.498007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.497834 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.498007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.497928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.498007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.497959 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.498264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.498122 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.498571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.498540 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.498654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.498608 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\") pod \"865040ec-fa04-46b4-9074-2dc97d5842e1\" (UID: \"865040ec-fa04-46b4-9074-2dc97d5842e1\") " Oct 29 22:37:26.498873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.498852 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.499022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499001 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.499175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499121 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.499175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499147 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.499275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499199 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.499402 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpcz\" (UniqueName: \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.499556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499525 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7mv\" (UniqueName: \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.499646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:26.499837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499819 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499842 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865040ec-fa04-46b4-9074-2dc97d5842e1-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499855 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499871 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499896 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499911 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.499934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499924 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjcl\" (UniqueName: \"kubernetes.io/projected/865040ec-fa04-46b4-9074-2dc97d5842e1-kube-api-access-sfjcl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.500341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.499938 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.504339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.504313 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy" pod "865040ec-fa04-46b4-9074-2dc97d5842e1" (UID: "865040ec-fa04-46b4-9074-2dc97d5842e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:26.514494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.514469 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx"] Oct 29 22:37:26.524663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.524634 4543 generic.go:356] "Generic (PLEG): container finished" podID="865040ec-fa04-46b4-9074-2dc97d5842e1" containerID="3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500" exitCode=0 Oct 29 22:37:26.524808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.524745 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" event={"ID":"865040ec-fa04-46b4-9074-2dc97d5842e1","Type":"ContainerDied","Data":"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500"} Oct 29 22:37:26.524808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.524790 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" event={"ID":"865040ec-fa04-46b4-9074-2dc97d5842e1","Type":"ContainerDied","Data":"208e5b330f45b285c44696062d2d0ba5b6c634e719c0c6cf869622132a0d4f6e"} Oct 29 22:37:26.524955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.524812 4543 scope.go:122] "RemoveContainer" containerID="3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500" Oct 29 22:37:26.524955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.524941 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8" Oct 29 22:37:26.552048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.552028 4543 scope.go:122] "RemoveContainer" containerID="3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500" Oct 29 22:37:26.552440 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:26.552418 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500\": container with ID starting with 3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500 not found: ID does not exist" containerID="3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500" Oct 29 22:37:26.552520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.552449 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500"} err="failed to get container status \"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500\": rpc error: code = NotFound desc = could not find container \"3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500\": container with ID starting with 3c68150a67be67e3736c93dc3e7edbd66aaf7de255b3e63dd82d3c3c36dd7500 not found: ID does not exist" Oct 29 22:37:26.582135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.582108 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:37:26.582390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.582375 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:37:26.601257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601230 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.601407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601302 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.601407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.601529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601462 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpcz\" (UniqueName: \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.601529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601519 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7mv\" (UniqueName: \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.601708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.601691 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865040ec-fa04-46b4-9074-2dc97d5842e1-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:26.602036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.602010 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.602520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.602498 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.604592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.604573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.612163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.612143 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:37:26.619493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.619475 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9ed94-openshift-default-674f58c4bf-r4nn8"] Oct 29 22:37:26.627829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.627809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7mv\" (UniqueName: \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\") pod \"maas-api-revoke-keys-e2e-inf-a-52429a-8njcx\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:26.627911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.627818 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpcz\" (UniqueName: \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\") pod \"payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.775737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.775719 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:26.815427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:26.815400 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:27.042683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.042608 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8"] Oct 29 22:37:27.048119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.048068 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab99c2c-96mk8"] Oct 29 22:37:27.097449 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:27.097404 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb213f6c4_c1cf_4232_8b83_4d905a24e8f1.slice/crio-892a6642a1186c4d69554a400ce49a21b97b2758825af31a69ebf24279d111ff WatchSource:0}: Error finding container 892a6642a1186c4d69554a400ce49a21b97b2758825af31a69ebf24279d111ff: Status 404 returned error can't find the container with id 892a6642a1186c4d69554a400ce49a21b97b2758825af31a69ebf24279d111ff Oct 29 22:37:27.098528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.098500 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:37:27.157932 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:27.157903 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f03a982_1be7_4084_b78b_a37a2cf5e70a.slice/crio-433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d WatchSource:0}: Error finding container 433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d: Status 404 returned error can't find the container with id 433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d Oct 29 22:37:27.158057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.157984 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx"] Oct 29 22:37:27.536322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.536291 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" event={"ID":"8f03a982-1be7-4084-b78b-a37a2cf5e70a","Type":"ContainerStarted","Data":"433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d"} Oct 29 22:37:27.539249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.539222 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" event={"ID":"b213f6c4-c1cf-4232-8b83-4d905a24e8f1","Type":"ContainerStarted","Data":"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd"} Oct 29 22:37:27.539385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.539278 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:27.539385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.539305 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" event={"ID":"b213f6c4-c1cf-4232-8b83-4d905a24e8f1","Type":"ContainerStarted","Data":"892a6642a1186c4d69554a400ce49a21b97b2758825af31a69ebf24279d111ff"} Oct 29 22:37:27.588057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:27.588012 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" podStartSLOduration=10.588000197 podStartE2EDuration="10.588000197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:27.583357361 +0000 UTC m=+3309.325288035" watchObservedRunningTime="2026-10-29 22:37:27.588000197 +0000 UTC m=+3309.329930871" Oct 29 22:37:27.962648 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:27.962614 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:37:28.304680 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:28.304652 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f03a982_1be7_4084_b78b_a37a2cf5e70a.slice/crio-conmon-472e79b1594ed67924cf03815670103bec05da7447d2f6332134c1deae942c3d.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:37:28.389755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:28.389724 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c5b2f1-1687-4532-8d87-75bd22a06818" path="/var/lib/kubelet/pods/83c5b2f1-1687-4532-8d87-75bd22a06818/volumes" Oct 29 22:37:28.390258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:28.390244 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865040ec-fa04-46b4-9074-2dc97d5842e1" path="/var/lib/kubelet/pods/865040ec-fa04-46b4-9074-2dc97d5842e1/volumes" Oct 29 22:37:28.550676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:28.550608 4543 generic.go:356] "Generic (PLEG): container finished" podID="8f03a982-1be7-4084-b78b-a37a2cf5e70a" containerID="472e79b1594ed67924cf03815670103bec05da7447d2f6332134c1deae942c3d" exitCode=0 Oct 29 22:37:28.551067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:28.550692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" event={"ID":"8f03a982-1be7-4084-b78b-a37a2cf5e70a","Type":"ContainerDied","Data":"472e79b1594ed67924cf03815670103bec05da7447d2f6332134c1deae942c3d"} Oct 29 22:37:29.750533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.750511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:29.761378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.761350 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\" (UniqueName: \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\") pod \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " Oct 29 22:37:29.761544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.761526 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\") pod \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " Oct 29 22:37:29.761612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.761601 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\") pod \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\" (UID: \"8f03a982-1be7-4084-b78b-a37a2cf5e70a\") " Oct 29 22:37:29.761866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.761842 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca" pod "8f03a982-1be7-4084-b78b-a37a2cf5e70a" (UID: "8f03a982-1be7-4084-b78b-a37a2cf5e70a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:29.762311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.762291 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:29.764352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.764326 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca" pod "8f03a982-1be7-4084-b78b-a37a2cf5e70a" (UID: "8f03a982-1be7-4084-b78b-a37a2cf5e70a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:29.764503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.764480 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv" pod "8f03a982-1be7-4084-b78b-a37a2cf5e70a" (UID: "8f03a982-1be7-4084-b78b-a37a2cf5e70a"). InnerVolumeSpecName "kube-api-access-zm7mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:29.863790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.863713 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f03a982-1be7-4084-b78b-a37a2cf5e70a-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:29.863790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:29.863741 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7mv\" (UniqueName: \"kubernetes.io/projected/8f03a982-1be7-4084-b78b-a37a2cf5e70a-kube-api-access-zm7mv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:30.573356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:30.573322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" Oct 29 22:37:30.573356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:30.573342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx" event={"ID":"8f03a982-1be7-4084-b78b-a37a2cf5e70a","Type":"ContainerDied","Data":"433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d"} Oct 29 22:37:30.573747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:30.573390 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433935d4f54402c068c9430f76b4f8e1e90d720334659930b98ff85ca2255f3d" Oct 29 22:37:31.557790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:31.557760 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:37:36.558620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:36.558568 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" containerID="cri-o://94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61" gracePeriod=25 Oct 29 22:37:36.673732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:36.673700 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:37:36.713232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:36.713206 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:37:36.895706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:36.895623 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:37.688690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:37.688652 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:37:37.735158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:37.735128 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:37:38.591337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:38.591305 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:37:41.675009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:41.674963 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" podUID="c43fb9b5-8654-4ed1-985e-8109c320e110" containerName="payload-processing" containerID="cri-o://493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a" gracePeriod=25 Oct 29 22:37:41.714379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:41.714346 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" podUID="7171c580-4a79-47a3-8662-69bfc3378505" containerName="payload-pre-processing" containerID="cri-o://10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd" gracePeriod=25 Oct 29 22:37:42.049677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.049650 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:37:42.071121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.071096 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\") pod \"c43fb9b5-8654-4ed1-985e-8109c320e110\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " Oct 29 22:37:42.071393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.071370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\" (UniqueName: \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\") pod \"c43fb9b5-8654-4ed1-985e-8109c320e110\" (UID: \"c43fb9b5-8654-4ed1-985e-8109c320e110\") " Oct 29 22:37:42.071526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.071441 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume" pod "c43fb9b5-8654-4ed1-985e-8109c320e110" (UID: "c43fb9b5-8654-4ed1-985e-8109c320e110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:42.072317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.072190 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c43fb9b5-8654-4ed1-985e-8109c320e110-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:42.074359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.074321 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp" pod "c43fb9b5-8654-4ed1-985e-8109c320e110" (UID: "c43fb9b5-8654-4ed1-985e-8109c320e110"). InnerVolumeSpecName "kube-api-access-458kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:42.080883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.080864 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:37:42.173114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.173063 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\") pod \"7171c580-4a79-47a3-8662-69bfc3378505\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " Oct 29 22:37:42.173301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.173167 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\" (UniqueName: \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\") pod \"7171c580-4a79-47a3-8662-69bfc3378505\" (UID: \"7171c580-4a79-47a3-8662-69bfc3378505\") " Oct 29 22:37:42.173458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.173433 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume" pod "7171c580-4a79-47a3-8662-69bfc3378505" (UID: "7171c580-4a79-47a3-8662-69bfc3378505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:42.173827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.173810 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7171c580-4a79-47a3-8662-69bfc3378505-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:42.173873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.173831 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458kp\" (UniqueName: \"kubernetes.io/projected/c43fb9b5-8654-4ed1-985e-8109c320e110-kube-api-access-458kp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:42.175385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.175362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj" pod "7171c580-4a79-47a3-8662-69bfc3378505" (UID: "7171c580-4a79-47a3-8662-69bfc3378505"). InnerVolumeSpecName "kube-api-access-x8jhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:42.274975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.274946 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jhj\" (UniqueName: \"kubernetes.io/projected/7171c580-4a79-47a3-8662-69bfc3378505-kube-api-access-x8jhj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:42.551083 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.551011 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:37:42.551981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.551966 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8f03a982-1be7-4084-b78b-a37a2cf5e70a" containerName="revoke-keys" Oct 29 22:37:42.552035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.551983 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f03a982-1be7-4084-b78b-a37a2cf5e70a" containerName="revoke-keys" Oct 29 22:37:42.552035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552009 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7171c580-4a79-47a3-8662-69bfc3378505" containerName="payload-pre-processing" Oct 29 22:37:42.552035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552015 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171c580-4a79-47a3-8662-69bfc3378505" containerName="payload-pre-processing" Oct 29 22:37:42.552035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552035 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c43fb9b5-8654-4ed1-985e-8109c320e110" containerName="payload-processing" Oct 29 22:37:42.552210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552040 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43fb9b5-8654-4ed1-985e-8109c320e110" containerName="payload-processing" Oct 29 22:37:42.552210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552200 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171c580-4a79-47a3-8662-69bfc3378505" containerName="payload-pre-processing" Oct 29 22:37:42.552284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552212 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43fb9b5-8654-4ed1-985e-8109c320e110" containerName="payload-processing" Oct 29 22:37:42.552284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.552227 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f03a982-1be7-4084-b78b-a37a2cf5e70a" containerName="revoke-keys" Oct 29 22:37:42.556310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.556294 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.586322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.585306 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-27b6db-serving-cert\"" Oct 29 22:37:42.592560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.592533 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:37:42.680464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.680434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.680464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.680464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z845z\" (UniqueName: \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.680933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.680613 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.680933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.680771 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.689884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.689851 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" podUID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" containerName="payload-processing" containerID="cri-o://e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766" gracePeriod=25 Oct 29 22:37:42.767337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.739403 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" podUID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" containerName="payload-pre-processing" containerID="cri-o://79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459" gracePeriod=25 Oct 29 22:37:42.789845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.782413 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.789845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.782590 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.789845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.782621 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z845z\" (UniqueName: \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.789845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.782767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.801497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.800571 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.833348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.833308 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.843691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.843185 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.854740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.849048 4543 generic.go:356] "Generic (PLEG): container finished" podID="7171c580-4a79-47a3-8662-69bfc3378505" containerID="10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd" exitCode=0 Oct 29 22:37:42.854740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.849448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" event={"ID":"7171c580-4a79-47a3-8662-69bfc3378505","Type":"ContainerDied","Data":"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd"} Oct 29 22:37:42.854740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.849509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" event={"ID":"7171c580-4a79-47a3-8662-69bfc3378505","Type":"ContainerDied","Data":"4e2c66f582b4be6051d626b60ce9140ab65408632c5b1e2b8b0df0ce070728f8"} Oct 29 22:37:42.854740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.849538 4543 scope.go:122] "RemoveContainer" containerID="10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd" Oct 29 22:37:42.854740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.849691 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh" Oct 29 22:37:42.858275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.858247 4543 generic.go:356] "Generic (PLEG): container finished" podID="c43fb9b5-8654-4ed1-985e-8109c320e110" containerID="493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a" exitCode=0 Oct 29 22:37:42.858399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.858297 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" event={"ID":"c43fb9b5-8654-4ed1-985e-8109c320e110","Type":"ContainerDied","Data":"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a"} Oct 29 22:37:42.858399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.858340 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" event={"ID":"c43fb9b5-8654-4ed1-985e-8109c320e110","Type":"ContainerDied","Data":"910a42f51fa77565854df2aab86e3d47032d1315fb779f302783e8a2e8728ee0"} Oct 29 22:37:42.858792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.858770 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx" Oct 29 22:37:42.861426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.861381 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z845z\" (UniqueName: \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\") pod \"maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.871725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.871700 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:42.959501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.959476 4543 scope.go:122] "RemoveContainer" containerID="10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd" Oct 29 22:37:42.967410 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:42.966545 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd\": container with ID starting with 10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd not found: ID does not exist" containerID="10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd" Oct 29 22:37:42.967410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.966588 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd"} err="failed to get container status \"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd\": rpc error: code = NotFound desc = could not find container \"10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd\": container with ID starting with 10d3505984354d1a7a61975aee73328de77c5ef9e3390c2fa925ad7d76171acd not found: ID does not exist" Oct 29 22:37:42.967410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.966612 4543 scope.go:122] "RemoveContainer" containerID="493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a" Oct 29 22:37:42.991254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:42.991228 4543 scope.go:122] "RemoveContainer" containerID="493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a" Oct 29 22:37:43.011980 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:43.011393 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a\": container with ID starting with 493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a not found: ID does not exist" containerID="493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a" Oct 29 22:37:43.011980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.011434 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a"} err="failed to get container status \"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a\": rpc error: code = NotFound desc = could not find container \"493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a\": container with ID starting with 493e29c43a49340d093da6cefd56c735c1ee9c8ff9fe6af2f3f24341d267730a not found: ID does not exist" Oct 29 22:37:43.012231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.012207 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:37:43.025666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.025641 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52429a-75f699f78b-vlhqx"] Oct 29 22:37:43.050805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.050775 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:37:43.056173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.056152 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52429a-57db4745f-xmnlh"] Oct 29 22:37:43.065832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.065807 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:37:43.088165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.088138 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\") pod \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " Oct 29 22:37:43.088461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.088443 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\" (UniqueName: \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\") pod \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\" (UID: \"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239\") " Oct 29 22:37:43.088606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.088580 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume" pod "7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" (UID: "7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:43.089262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.089230 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:43.091279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.091253 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4" pod "7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" (UID: "7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239"). InnerVolumeSpecName "kube-api-access-dllz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:43.190572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.190542 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllz4\" (UniqueName: \"kubernetes.io/projected/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239-kube-api-access-dllz4\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:43.199654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.199632 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:37:43.202394 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:43.202369 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf56622f_3488_40de_8962_13187f47e0d4.slice/crio-a7bc7dc221621dccd1deab965a6659734bcb505c59c580cc943a4e7b13051627 WatchSource:0}: Error finding container a7bc7dc221621dccd1deab965a6659734bcb505c59c580cc943a4e7b13051627: Status 404 returned error can't find the container with id a7bc7dc221621dccd1deab965a6659734bcb505c59c580cc943a4e7b13051627 Oct 29 22:37:43.202489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.202389 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:37:43.203854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.203836 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:37:43.291682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.291654 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\" (UniqueName: \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\") pod \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " Oct 29 22:37:43.291920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.291896 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\") pod \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\" (UID: \"de465347-1a0b-4c58-9a53-6c78fb2e8b19\") " Oct 29 22:37:43.292423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.292396 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume" pod "de465347-1a0b-4c58-9a53-6c78fb2e8b19" (UID: "de465347-1a0b-4c58-9a53-6c78fb2e8b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:43.292821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.292793 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de465347-1a0b-4c58-9a53-6c78fb2e8b19-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:43.295150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.295121 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t" pod "de465347-1a0b-4c58-9a53-6c78fb2e8b19" (UID: "de465347-1a0b-4c58-9a53-6c78fb2e8b19"). InnerVolumeSpecName "kube-api-access-wr77t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:43.393923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:43.393870 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr77t\" (UniqueName: \"kubernetes.io/projected/de465347-1a0b-4c58-9a53-6c78fb2e8b19-kube-api-access-wr77t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:44.060252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.060163 4543 generic.go:356] "Generic (PLEG): container finished" podID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" containerID="79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459" exitCode=0 Oct 29 22:37:44.060252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.060239 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" event={"ID":"de465347-1a0b-4c58-9a53-6c78fb2e8b19","Type":"ContainerDied","Data":"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459"} Oct 29 22:37:44.060771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.060314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" event={"ID":"de465347-1a0b-4c58-9a53-6c78fb2e8b19","Type":"ContainerDied","Data":"6f3b40a242be20630e86d4573b9265b76ac76ade4e7f5f8ff22d9bca7257bbef"} Oct 29 22:37:44.060771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.060322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4" Oct 29 22:37:44.060771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.060345 4543 scope.go:122] "RemoveContainer" containerID="79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459" Oct 29 22:37:44.062945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.062916 4543 generic.go:356] "Generic (PLEG): container finished" podID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" containerID="e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766" exitCode=0 Oct 29 22:37:44.063050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.063002 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" event={"ID":"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239","Type":"ContainerDied","Data":"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766"} Oct 29 22:37:44.063050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.063033 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" event={"ID":"7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239","Type":"ContainerDied","Data":"a395af450ae9c1940beedc5a041b108dcc42b5efce2977843f09dbdcfe383091"} Oct 29 22:37:44.063222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.063169 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj" Oct 29 22:37:44.067492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.067466 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" event={"ID":"bf56622f-3488-40de-8962-13187f47e0d4","Type":"ContainerStarted","Data":"bc9bcf54ba0575775d86902782d159d44cdef1ac9b204557a6aa6a0e58a88615"} Oct 29 22:37:44.067626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.067513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" event={"ID":"bf56622f-3488-40de-8962-13187f47e0d4","Type":"ContainerStarted","Data":"a7bc7dc221621dccd1deab965a6659734bcb505c59c580cc943a4e7b13051627"} Oct 29 22:37:44.067626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.067563 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:44.081488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.081473 4543 scope.go:122] "RemoveContainer" containerID="79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459" Oct 29 22:37:44.081780 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:44.081763 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459\": container with ID starting with 79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459 not found: ID does not exist" containerID="79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459" Oct 29 22:37:44.081860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.081787 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459"} err="failed to get container status \"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459\": rpc error: code = NotFound desc = could not find container \"79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459\": container with ID starting with 79d98d8f4e359215755340244c941985916043b4feece0a173609da784a34459 not found: ID does not exist" Oct 29 22:37:44.081860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.081803 4543 scope.go:122] "RemoveContainer" containerID="e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766" Oct 29 22:37:44.094720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.094701 4543 scope.go:122] "RemoveContainer" containerID="e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766" Oct 29 22:37:44.095063 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:44.095043 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766\": container with ID starting with e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766 not found: ID does not exist" containerID="e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766" Oct 29 22:37:44.095235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.095069 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766"} err="failed to get container status \"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766\": rpc error: code = NotFound desc = could not find container \"e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766\": container with ID starting with e571837bb931e52e45a109a33a91b89b3baece5b4b839e08e67d92298e913766 not found: ID does not exist" Oct 29 22:37:44.147536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.147495 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" podStartSLOduration=1.7884673869999999 podStartE2EDuration="2.147483174s" totalImagesPullingTime="359.015787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:44.141699666 +0000 UTC m=+3325.883630339" watchObservedRunningTime="2026-10-29 22:37:44.147483174 +0000 UTC m=+3325.889413848" Oct 29 22:37:44.185980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.185959 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:37:44.191490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.191468 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75ea0c-dc9b5ccb6-686hj"] Oct 29 22:37:44.231003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.230968 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:37:44.236842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.236822 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75ea0c-c494c8f67-rcrj4"] Oct 29 22:37:44.389846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.389764 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7171c580-4a79-47a3-8662-69bfc3378505" path="/var/lib/kubelet/pods/7171c580-4a79-47a3-8662-69bfc3378505/volumes" Oct 29 22:37:44.390258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.390240 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" path="/var/lib/kubelet/pods/7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239/volumes" Oct 29 22:37:44.390682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.390665 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43fb9b5-8654-4ed1-985e-8109c320e110" path="/var/lib/kubelet/pods/c43fb9b5-8654-4ed1-985e-8109c320e110/volumes" Oct 29 22:37:44.391089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:44.391058 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" path="/var/lib/kubelet/pods/de465347-1a0b-4c58-9a53-6c78fb2e8b19/volumes" Oct 29 22:37:45.291849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:45.291813 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln"] Oct 29 22:37:45.304295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:45.304265 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-219ccc-4x9ln"] Oct 29 22:37:46.396987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:46.396950 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc72caf-1d1d-40e4-a261-c242b0287762" path="/var/lib/kubelet/pods/5fc72caf-1d1d-40e4-a261-c242b0287762/volumes" Oct 29 22:37:46.895680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:46.895640 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:49.979591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:49.974368 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx"] Oct 29 22:37:50.016194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:50.016158 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52429a-8njcx"] Oct 29 22:37:50.093416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:50.093389 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:37:50.388971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:50.388890 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f03a982-1be7-4084-b78b-a37a2cf5e70a" path="/var/lib/kubelet/pods/8f03a982-1be7-4084-b78b-a37a2cf5e70a/volumes" Oct 29 22:37:52.767743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.767706 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:37:52.769224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769200 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" containerName="payload-processing" Oct 29 22:37:52.769417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769392 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" containerName="payload-processing" Oct 29 22:37:52.769561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769480 4543 cpu_manager.go:414] "containerMap: removing container" podUID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" containerName="payload-pre-processing" Oct 29 22:37:52.769561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769492 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" containerName="payload-pre-processing" Oct 29 22:37:52.769801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769777 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf5d693-1097-4cbb-a0b5-e8ecb6e3e239" containerName="payload-processing" Oct 29 22:37:52.769921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.769808 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="de465347-1a0b-4c58-9a53-6c78fb2e8b19" containerName="payload-pre-processing" Oct 29 22:37:52.774814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.774792 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.811491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.811464 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52429a-openshift-default-dockercfg-rbvt2\"" Oct 29 22:37:52.820656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.820631 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:37:52.896302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896233 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzcj\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896617 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896665 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896777 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896830 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.896901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.896892 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.897031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.897010 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998278 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzcj\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998314 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998451 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998638 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998707 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998730 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998772 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998881 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998968 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.998993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.998973 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.999158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.999048 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:52.999480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:52.999457 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.001356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.001333 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.001501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.001483 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.024564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.024505 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.024651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.024643 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzcj\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.098430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.098410 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:53.505179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.503959 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:53.505179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.504051 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:53.505179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.504125 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:37:53.594114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:53.587195 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:37:54.232062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:54.232019 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" event={"ID":"8821d567-1a13-4c2d-bed7-39061baa3e88","Type":"ContainerStarted","Data":"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34"} Oct 29 22:37:54.232472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:54.232101 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" event={"ID":"8821d567-1a13-4c2d-bed7-39061baa3e88","Type":"ContainerStarted","Data":"2a81ff2013a7ba4ba77dc9c0431aefb5326f81bb8985533902d2f373f2127da3"} Oct 29 22:37:54.286043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:54.285993 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podStartSLOduration=2.285979958 podStartE2EDuration="2.285979958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:54.283719678 +0000 UTC m=+3336.025650352" watchObservedRunningTime="2026-10-29 22:37:54.285979958 +0000 UTC m=+3336.027910631" Oct 29 22:37:55.100099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:55.099274 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:37:55.101462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:55.101434 4543 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" start-of-body= Oct 29 22:37:55.101606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:55.101487 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" Oct 29 22:37:56.098952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.098917 4543 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" start-of-body= Oct 29 22:37:56.099387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.098989 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" Oct 29 22:37:56.907350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.907321 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:37:56.981035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.980990 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\" (UniqueName: \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\") pod \"faeb36ce-ec09-40a5-a268-9d258a946bd1\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " Oct 29 22:37:56.981272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.981253 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\") pod \"faeb36ce-ec09-40a5-a268-9d258a946bd1\" (UID: \"faeb36ce-ec09-40a5-a268-9d258a946bd1\") " Oct 29 22:37:56.981828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.981701 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume" pod "faeb36ce-ec09-40a5-a268-9d258a946bd1" (UID: "faeb36ce-ec09-40a5-a268-9d258a946bd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:56.982339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.982303 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faeb36ce-ec09-40a5-a268-9d258a946bd1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:56.988524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:56.988467 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d" pod "faeb36ce-ec09-40a5-a268-9d258a946bd1" (UID: "faeb36ce-ec09-40a5-a268-9d258a946bd1"). InnerVolumeSpecName "kube-api-access-v5s9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:57.084138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.084099 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5s9d\" (UniqueName: \"kubernetes.io/projected/faeb36ce-ec09-40a5-a268-9d258a946bd1-kube-api-access-v5s9d\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:37:57.098951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.098921 4543 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" start-of-body= Oct 29 22:37:57.099174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.098989 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" Oct 29 22:37:57.326477 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.326440 4543 generic.go:356] "Generic (PLEG): container finished" podID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerID="94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61" exitCode=0 Oct 29 22:37:57.326864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.326843 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" Oct 29 22:37:57.327091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.327049 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" event={"ID":"faeb36ce-ec09-40a5-a268-9d258a946bd1","Type":"ContainerDied","Data":"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61"} Oct 29 22:37:57.327189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.327136 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" event={"ID":"faeb36ce-ec09-40a5-a268-9d258a946bd1","Type":"ContainerDied","Data":"4a307610a07d5531b31225bdfed7d5d14520f94b854f2c83b5bc2cb8333bda14"} Oct 29 22:37:57.327189 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.327159 4543 scope.go:122] "RemoveContainer" containerID="94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61" Oct 29 22:37:57.371681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.371660 4543 scope.go:122] "RemoveContainer" containerID="94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61" Oct 29 22:37:57.371979 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:57.371959 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61\": container with ID starting with 94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61 not found: ID does not exist" containerID="94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61" Oct 29 22:37:57.372057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.371992 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61"} err="failed to get container status \"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61\": rpc error: code = NotFound desc = could not find container \"94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61\": container with ID starting with 94b2cdf276bf3979f84b0290d6e8b63f0370315ebfc73af9bad2776205c3be61 not found: ID does not exist" Oct 29 22:37:57.413356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.413331 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:37:57.418318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.418299 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh"] Oct 29 22:37:57.894225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:57.894183 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-adfb9d-59776ff6ff-zxjdh" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.81:50052\" within 1s: context deadline exceeded" Oct 29 22:37:57.894399 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:37:57.894272 4543 logging.go:55] [core] [Channel #27533 SubChannel #27534]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.81:50052", ServerName: "10.133.1.81:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.81:50052: operation was canceled" Oct 29 22:37:57.981992 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:37:57.981947 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:37:58.099421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:58.099387 4543 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" start-of-body= Oct 29 22:37:58.099795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:58.099434 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.167:15021/healthz/ready\": dial tcp 10.133.1.167:15021: connect: connection refused" Oct 29 22:37:58.261365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:58.261323 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:37:58.261621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:58.261592 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" containerID="cri-o://665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34" gracePeriod=30 Oct 29 22:37:58.556155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:37:58.544004 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" path="/var/lib/kubelet/pods/faeb36ce-ec09-40a5-a268-9d258a946bd1/volumes" Oct 29 22:38:00.307224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.307184 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc"] Oct 29 22:38:00.308255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.308234 4543 cpu_manager.go:414] "containerMap: removing container" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" Oct 29 22:38:00.308255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.308252 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" Oct 29 22:38:00.308483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.308423 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeb36ce-ec09-40a5-a268-9d258a946bd1" containerName="payload-pre-processing" Oct 29 22:38:00.314152 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.314136 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.360660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.360626 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:38:00.361503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.361461 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:38:00.365465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.365434 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc"] Oct 29 22:38:00.417191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.417161 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.417625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.417600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.417862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.417841 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gptqg\" (UniqueName: \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.519868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.519824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.520025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.520010 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.520136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.520058 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gptqg\" (UniqueName: \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.520520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.520502 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.522608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.522587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.562136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.561998 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gptqg\" (UniqueName: \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.660859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.660819 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:00.983585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:00.983553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc"] Oct 29 22:38:01.530180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:01.530138 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" event={"ID":"95aeb93e-cb6a-4179-8bf2-2fc034e11a52","Type":"ContainerStarted","Data":"c8d924a2a8865e54df4b86c8ae0af8551fed86265aeba0bec20f41f78f6edb9f"} Oct 29 22:38:02.538491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:02.538457 4543 generic.go:356] "Generic (PLEG): container finished" podID="95aeb93e-cb6a-4179-8bf2-2fc034e11a52" containerID="4ec75d1f5f4ca1dfb829c614eb72ae971f5dffbe7216e74aa8576ca04557192b" exitCode=0 Oct 29 22:38:02.538874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:02.538533 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" event={"ID":"95aeb93e-cb6a-4179-8bf2-2fc034e11a52","Type":"ContainerDied","Data":"4ec75d1f5f4ca1dfb829c614eb72ae971f5dffbe7216e74aa8576ca04557192b"} Oct 29 22:38:03.807162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.807136 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:03.902213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.902142 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\") pod \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " Oct 29 22:38:03.902382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.902248 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\" (UniqueName: \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\") pod \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " Oct 29 22:38:03.902467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.902413 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\") pod \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\" (UID: \"95aeb93e-cb6a-4179-8bf2-2fc034e11a52\") " Oct 29 22:38:03.902467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.902450 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca" pod "95aeb93e-cb6a-4179-8bf2-2fc034e11a52" (UID: "95aeb93e-cb6a-4179-8bf2-2fc034e11a52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:03.903219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.903195 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:03.904599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.904569 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca" pod "95aeb93e-cb6a-4179-8bf2-2fc034e11a52" (UID: "95aeb93e-cb6a-4179-8bf2-2fc034e11a52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:03.904726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:03.904703 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg" pod "95aeb93e-cb6a-4179-8bf2-2fc034e11a52" (UID: "95aeb93e-cb6a-4179-8bf2-2fc034e11a52"). InnerVolumeSpecName "kube-api-access-gptqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:04.004595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:04.004565 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gptqg\" (UniqueName: \"kubernetes.io/projected/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-kube-api-access-gptqg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:04.004595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:04.004595 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb93e-cb6a-4179-8bf2-2fc034e11a52-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:04.580969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:04.580937 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" Oct 29 22:38:04.581140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:04.580959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc" event={"ID":"95aeb93e-cb6a-4179-8bf2-2fc034e11a52","Type":"ContainerDied","Data":"c8d924a2a8865e54df4b86c8ae0af8551fed86265aeba0bec20f41f78f6edb9f"} Oct 29 22:38:04.581140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:04.581016 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8d924a2a8865e54df4b86c8ae0af8551fed86265aeba0bec20f41f78f6edb9f" Oct 29 22:38:05.863376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:05.863343 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:38:07.289495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.289437 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:38:07.436802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.436756 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.437139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.437114 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.443201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.443167 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.443451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.443428 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.453006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.443817 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.453639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.453607 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.462899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.462866 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.464723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.464693 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.466878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.464996 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\") pod \"8821d567-1a13-4c2d-bed7-39061baa3e88\" (UID: \"8821d567-1a13-4c2d-bed7-39061baa3e88\") " Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.446983 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.447384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.447809 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.448196 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.463166 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468572 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468866 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468887 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468899 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8821d567-1a13-4c2d-bed7-39061baa3e88-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468922 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.469040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.468931 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.469779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.469666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:07.470987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.470951 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "kube-api-access-htzcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:07.473652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.471362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo" pod "8821d567-1a13-4c2d-bed7-39061baa3e88" (UID: "8821d567-1a13-4c2d-bed7-39061baa3e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:07.570138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.570089 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.570138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.570111 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.570138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.570120 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzcj\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-kube-api-access-htzcj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.570138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.570131 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8821d567-1a13-4c2d-bed7-39061baa3e88-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:07.690796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.690766 4543 generic.go:356] "Generic (PLEG): container finished" podID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerID="665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34" exitCode=0 Oct 29 22:38:07.690913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.690814 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" Oct 29 22:38:07.690913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.690854 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" event={"ID":"8821d567-1a13-4c2d-bed7-39061baa3e88","Type":"ContainerDied","Data":"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34"} Oct 29 22:38:07.690913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.690907 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv" event={"ID":"8821d567-1a13-4c2d-bed7-39061baa3e88","Type":"ContainerDied","Data":"2a81ff2013a7ba4ba77dc9c0431aefb5326f81bb8985533902d2f373f2127da3"} Oct 29 22:38:07.691096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.690934 4543 scope.go:122] "RemoveContainer" containerID="665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34" Oct 29 22:38:07.704037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.704018 4543 scope.go:122] "RemoveContainer" containerID="665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34" Oct 29 22:38:07.704285 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:07.704267 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34\": container with ID starting with 665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34 not found: ID does not exist" containerID="665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34" Oct 29 22:38:07.704336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.704291 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34"} err="failed to get container status \"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34\": rpc error: code = NotFound desc = could not find container \"665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34\": container with ID starting with 665a002726cf24ba75da612b391e6f7badda98e92191c545bbf5418922162c34 not found: ID does not exist" Oct 29 22:38:07.759670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.759648 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:38:07.772562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:07.772540 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-k6wqv"] Oct 29 22:38:08.390473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:08.390442 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" path="/var/lib/kubelet/pods/8821d567-1a13-4c2d-bed7-39061baa3e88/volumes" Oct 29 22:38:10.837412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:10.837379 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:38:10.864046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:10.864010 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" containerID="cri-o://b44d6b6e33052af8c5fa721c17c5840e708fbb246fcef7bda5d008e0391d7a34" gracePeriod=25 Oct 29 22:38:13.847607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:13.847573 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:38:13.848094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:13.847868 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerName="maas-api" containerID="cri-o://be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b" gracePeriod=30 Oct 29 22:38:14.723133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.721320 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:38:14.816880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.816796 4543 generic.go:356] "Generic (PLEG): container finished" podID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerID="be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b" exitCode=0 Oct 29 22:38:14.817049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.816907 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" Oct 29 22:38:14.817049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.816988 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" event={"ID":"0434f25f-8dde-4881-9a86-dbe7d952cb3a","Type":"ContainerDied","Data":"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b"} Oct 29 22:38:14.817161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.817107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" event={"ID":"0434f25f-8dde-4881-9a86-dbe7d952cb3a","Type":"ContainerDied","Data":"b0499acb44c4fbedac4dd895acec51727f0a51234f09f039ffcb2790db60d986"} Oct 29 22:38:14.817161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.817145 4543 scope.go:122] "RemoveContainer" containerID="be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b" Oct 29 22:38:14.821561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.821532 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\") pod \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " Oct 29 22:38:14.822451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.822220 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\") pod \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " Oct 29 22:38:14.823656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.822908 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\" (UniqueName: \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\") pod \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " Oct 29 22:38:14.823656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.823034 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\") pod \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\" (UID: \"0434f25f-8dde-4881-9a86-dbe7d952cb3a\") " Oct 29 22:38:14.823656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.823611 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca" pod "0434f25f-8dde-4881-9a86-dbe7d952cb3a" (UID: "0434f25f-8dde-4881-9a86-dbe7d952cb3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:14.828666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.825202 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs" pod "0434f25f-8dde-4881-9a86-dbe7d952cb3a" (UID: "0434f25f-8dde-4881-9a86-dbe7d952cb3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:14.828666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.825340 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0434f25f-8dde-4881-9a86-dbe7d952cb3a-openshift-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:14.831514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.831394 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls" pod "0434f25f-8dde-4881-9a86-dbe7d952cb3a" (UID: "0434f25f-8dde-4881-9a86-dbe7d952cb3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:14.832330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.832310 4543 scope.go:122] "RemoveContainer" containerID="be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b" Oct 29 22:38:14.832659 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:14.832635 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b\": container with ID starting with be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b not found: ID does not exist" containerID="be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b" Oct 29 22:38:14.832749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.832666 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b"} err="failed to get container status \"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b\": rpc error: code = NotFound desc = could not find container \"be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b\": container with ID starting with be9670f3b94aac06a78b5f68eb6c7c70c5baea03587cab7a49432bbfbe00696b not found: ID does not exist" Oct 29 22:38:14.833033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.833003 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57" pod "0434f25f-8dde-4881-9a86-dbe7d952cb3a" (UID: "0434f25f-8dde-4881-9a86-dbe7d952cb3a"). InnerVolumeSpecName "kube-api-access-m8v57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:14.935103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.934838 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-metrics-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:14.935103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.934884 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0434f25f-8dde-4881-9a86-dbe7d952cb3a-maas-api-tls\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:14.935103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:14.934900 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v57\" (UniqueName: \"kubernetes.io/projected/0434f25f-8dde-4881-9a86-dbe7d952cb3a-kube-api-access-m8v57\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:15.201344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.201289 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:38:15.225719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.225683 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn"] Oct 29 22:38:15.296498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.296464 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:38:15.296696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.296675 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" containerID="cri-o://153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b" gracePeriod=30 Oct 29 22:38:15.298242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.298218 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:15.837799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:15.837764 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" containerID="cri-o://f14da10a47ca4c19f73a05f04b76a4e4b4516e5e47d2550183c3feb2e2c1e068" gracePeriod=25 Oct 29 22:38:16.390911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:16.390868 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" path="/var/lib/kubelet/pods/0434f25f-8dde-4881-9a86-dbe7d952cb3a/volumes" Oct 29 22:38:16.538742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:16.538711 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc"] Oct 29 22:38:16.579029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:16.578996 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-50d4af-2pcqc"] Oct 29 22:38:17.370701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:17.370662 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75ea0c-7496684f85-4mmzn" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.123:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:18.389091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:18.389042 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aeb93e-cb6a-4179-8bf2-2fc034e11a52" path="/var/lib/kubelet/pods/95aeb93e-cb6a-4179-8bf2-2fc034e11a52/volumes" Oct 29 22:38:19.659583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:19.659543 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:20.591882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.591856 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:38:20.680785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.680752 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.680840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.680876 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.680940 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.680974 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681108 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681156 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681281 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:20.681750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681430 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681498 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:20.681750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681520 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\") pod \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\" (UID: \"a89a9a05-ea8c-432f-90f9-df3f6bf20645\") " Oct 29 22:38:20.681750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:20.682034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.681938 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:20.682614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.682583 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.682741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.682617 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.682741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.682633 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.682741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.682645 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.682741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.682659 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.684680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.684642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:20.685059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.684953 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:20.685059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.685036 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:20.685397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.685373 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf" pod "a89a9a05-ea8c-432f-90f9-df3f6bf20645" (UID: "a89a9a05-ea8c-432f-90f9-df3f6bf20645"). InnerVolumeSpecName "kube-api-access-r9qdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:20.784866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.784829 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.784866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.784866 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.785172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.784881 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.785172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.784893 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/a89a9a05-ea8c-432f-90f9-df3f6bf20645-kube-api-access-r9qdf\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:20.883939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.883861 4543 generic.go:356] "Generic (PLEG): container finished" podID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerID="153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b" exitCode=0 Oct 29 22:38:20.883939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.883901 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" Oct 29 22:38:20.884215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.883938 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" event={"ID":"a89a9a05-ea8c-432f-90f9-df3f6bf20645","Type":"ContainerDied","Data":"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b"} Oct 29 22:38:20.884215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.883999 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf" event={"ID":"a89a9a05-ea8c-432f-90f9-df3f6bf20645","Type":"ContainerDied","Data":"e77d8a85ce48188f77d513181e1299fc1e8b3c5ba29c4c2751f93b71424deafe"} Oct 29 22:38:20.884215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.884032 4543 scope.go:122] "RemoveContainer" containerID="153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b" Oct 29 22:38:20.897563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.897541 4543 scope.go:122] "RemoveContainer" containerID="153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b" Oct 29 22:38:20.897817 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:20.897796 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b\": container with ID starting with 153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b not found: ID does not exist" containerID="153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b" Oct 29 22:38:20.897874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.897826 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b"} err="failed to get container status \"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b\": rpc error: code = NotFound desc = could not find container \"153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b\": container with ID starting with 153c345bf497f27fec791eb6ada4c34fbdf0301bafd1a690d828c521af7a2d8b not found: ID does not exist" Oct 29 22:38:20.947042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.947018 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:38:20.959950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:20.959930 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-tv5rf"] Oct 29 22:38:22.389492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:22.389456 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" path="/var/lib/kubelet/pods/a89a9a05-ea8c-432f-90f9-df3f6bf20645/volumes" Oct 29 22:38:24.328162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.328109 4543 scope.go:122] "RemoveContainer" containerID="76a5b5839c60eaff5dec858aba579d1dba6522ff8a12597de254bab1368dc170" Oct 29 22:38:24.342142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.342123 4543 scope.go:122] "RemoveContainer" containerID="1ef972edae1e1b936dc03be4304f162c911d966cfa84be9c7ee87b04c70e81c3" Oct 29 22:38:24.353683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.353636 4543 scope.go:122] "RemoveContainer" containerID="9b2752b2192f765370b684b5cfecc4eee929f93e8a43a6927c10bc2c8adbdd9d" Oct 29 22:38:24.364317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.364299 4543 scope.go:122] "RemoveContainer" containerID="c8dc343555b7f8978d280f2e76ae102c2bc35b661e50a4d3539062526b3ac0c5" Oct 29 22:38:24.375054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.375030 4543 scope.go:122] "RemoveContainer" containerID="a022030bd99bae4bb0f42f66e5620b4fe6ade5c8aa14ef1607c7fc82924ab428" Oct 29 22:38:24.385909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.385892 4543 scope.go:122] "RemoveContainer" containerID="ab127017f5d4ab0d79c4f8b91f50f6c8a81cd1d68d8d32e7223f0dd3e339e32c" Oct 29 22:38:24.395921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.395898 4543 scope.go:122] "RemoveContainer" containerID="d8e05d4076097dab1f022a17e1835a88e3ca378fee9a71f4a75c46456fc3f4e1" Oct 29 22:38:24.409574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.409539 4543 scope.go:122] "RemoveContainer" containerID="7d40fa9593ede4347fa981b1b51e9dfa52fa52775cc0521f753e72a663705426" Oct 29 22:38:24.419532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.419511 4543 scope.go:122] "RemoveContainer" containerID="1cd779d0c155221271e6b557f6835fb394acc1ebb5a34a8936754c5d502d17dd" Oct 29 22:38:24.533020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.532995 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh"] Oct 29 22:38:24.534059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534038 4543 cpu_manager.go:414] "containerMap: removing container" podUID="95aeb93e-cb6a-4179-8bf2-2fc034e11a52" containerName="revoke-keys" Oct 29 22:38:24.534173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534062 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aeb93e-cb6a-4179-8bf2-2fc034e11a52" containerName="revoke-keys" Oct 29 22:38:24.534173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534134 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" Oct 29 22:38:24.534173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534141 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" Oct 29 22:38:24.534173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534154 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerName="maas-api" Oct 29 22:38:24.534173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534159 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerName="maas-api" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534192 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534197 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534340 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89a9a05-ea8c-432f-90f9-df3f6bf20645" containerName="istio-proxy" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534350 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aeb93e-cb6a-4179-8bf2-2fc034e11a52" containerName="revoke-keys" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534372 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8821d567-1a13-4c2d-bed7-39061baa3e88" containerName="istio-proxy" Oct 29 22:38:24.534491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.534382 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0434f25f-8dde-4881-9a86-dbe7d952cb3a" containerName="maas-api" Oct 29 22:38:24.541535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.541520 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.564775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.564750 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh"] Oct 29 22:38:24.628468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.628403 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.628577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.628476 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.628694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.628675 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsb7\" (UniqueName: \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.631996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.631975 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:38:24.632163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.632148 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:38:24.730467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.730441 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.730603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.730502 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.730678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.730645 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsb7\" (UniqueName: \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.731260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.731203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.732998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.732970 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.757508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.757481 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsb7\" (UniqueName: \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\") pod \"maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:24.861723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:24.861700 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:25.179156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:25.179121 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh"] Oct 29 22:38:25.179531 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:38:25.179507 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abe43ab_a867_424e_9522_78453a81099c.slice/crio-ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65 WatchSource:0}: Error finding container ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65: Status 404 returned error can't find the container with id ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65 Oct 29 22:38:25.297739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:25.297704 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:25.951825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:25.951722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" event={"ID":"5abe43ab-a867-424e-9522-78453a81099c","Type":"ContainerStarted","Data":"6d2aaead9e90254b7e29f21e61c4d552a36458275953b1387bf9380225c51ecc"} Oct 29 22:38:25.951825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:25.951821 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" event={"ID":"5abe43ab-a867-424e-9522-78453a81099c","Type":"ContainerStarted","Data":"ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65"} Oct 29 22:38:25.996404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:25.996346 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" podStartSLOduration=1.996332417 podStartE2EDuration="1.996332417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:25.995320704 +0000 UTC m=+3367.737251376" watchObservedRunningTime="2026-10-29 22:38:25.996332417 +0000 UTC m=+3367.738263089" Oct 29 22:38:26.140005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.139978 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk"] Oct 29 22:38:26.145312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.145292 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.167696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.167642 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk"] Oct 29 22:38:26.190434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.190407 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.190581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.190485 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scx7\" (UniqueName: \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.190581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.190524 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.255739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.255714 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:38:26.255943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.255922 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" containerID="cri-o://feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33" gracePeriod=30 Oct 29 22:38:26.292202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.292167 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scx7\" (UniqueName: \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.292333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.292222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.292643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.292623 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.292794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.292771 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.295050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.295021 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.319993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.319974 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scx7\" (UniqueName: \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\") pod \"maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.461790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.461754 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:26.783799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.783772 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk"] Oct 29 22:38:26.784592 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:38:26.784563 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0b0800_15d6_44c8_9ebe_e0ce5a4fc1de.slice/crio-c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165 WatchSource:0}: Error finding container c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165: Status 404 returned error can't find the container with id c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165 Oct 29 22:38:26.961574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.961547 4543 generic.go:356] "Generic (PLEG): container finished" podID="5abe43ab-a867-424e-9522-78453a81099c" containerID="6d2aaead9e90254b7e29f21e61c4d552a36458275953b1387bf9380225c51ecc" exitCode=0 Oct 29 22:38:26.961914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.961636 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" event={"ID":"5abe43ab-a867-424e-9522-78453a81099c","Type":"ContainerDied","Data":"6d2aaead9e90254b7e29f21e61c4d552a36458275953b1387bf9380225c51ecc"} Oct 29 22:38:26.963264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:26.963240 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" event={"ID":"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de","Type":"ContainerStarted","Data":"c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165"} Oct 29 22:38:27.962903 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:27.962870 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:38:27.974066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:27.974042 4543 generic.go:356] "Generic (PLEG): container finished" podID="4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" containerID="83621820aeabe4eb26e70d0e4c5e8bf09db9b80d726acf56aac5471749a71a82" exitCode=0 Oct 29 22:38:27.974224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:27.974119 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" event={"ID":"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de","Type":"ContainerDied","Data":"83621820aeabe4eb26e70d0e4c5e8bf09db9b80d726acf56aac5471749a71a82"} Oct 29 22:38:28.151608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.151586 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:28.245619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.245593 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\") pod \"5abe43ab-a867-424e-9522-78453a81099c\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " Oct 29 22:38:28.245881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.245862 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\" (UniqueName: \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\") pod \"5abe43ab-a867-424e-9522-78453a81099c\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " Oct 29 22:38:28.245975 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.245905 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\") pod \"5abe43ab-a867-424e-9522-78453a81099c\" (UID: \"5abe43ab-a867-424e-9522-78453a81099c\") " Oct 29 22:38:28.246031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.245967 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca" pod "5abe43ab-a867-424e-9522-78453a81099c" (UID: "5abe43ab-a867-424e-9522-78453a81099c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:28.246754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.246728 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abe43ab-a867-424e-9522-78453a81099c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:28.248154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.248131 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca" pod "5abe43ab-a867-424e-9522-78453a81099c" (UID: "5abe43ab-a867-424e-9522-78453a81099c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:28.248251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.248153 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7" pod "5abe43ab-a867-424e-9522-78453a81099c" (UID: "5abe43ab-a867-424e-9522-78453a81099c"). InnerVolumeSpecName "kube-api-access-sgsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:28.348249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.348227 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abe43ab-a867-424e-9522-78453a81099c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:28.348249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.348249 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsb7\" (UniqueName: \"kubernetes.io/projected/5abe43ab-a867-424e-9522-78453a81099c-kube-api-access-sgsb7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:28.989734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.989697 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" Oct 29 22:38:28.989734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.989710 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh" event={"ID":"5abe43ab-a867-424e-9522-78453a81099c","Type":"ContainerDied","Data":"ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65"} Oct 29 22:38:28.990252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:28.989799 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3d2c6095236e23c0f93562a5d99cf05dd047ee9b474fac7bdbfd28c3d34c65" Oct 29 22:38:29.221280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.218049 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:38:29.221280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.218249 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:38:29.297221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.294282 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:29.396874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.396832 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\") pod \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " Oct 29 22:38:29.397636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.397506 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca" pod "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" (UID: "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:29.398459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.398063 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\" (UniqueName: \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\") pod \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " Oct 29 22:38:29.398684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.398667 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\") pod \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\" (UID: \"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de\") " Oct 29 22:38:29.401331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.401213 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.419651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.419606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca" pod "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" (UID: "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:29.433349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.433302 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7" pod "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" (UID: "4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de"). InnerVolumeSpecName "kube-api-access-5scx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:29.511587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.511555 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scx7\" (UniqueName: \"kubernetes.io/projected/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-kube-api-access-5scx7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.511856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.511836 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:29.666722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:29.666624 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:30.002673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:30.002635 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" event={"ID":"4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de","Type":"ContainerDied","Data":"c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165"} Oct 29 22:38:30.003063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:30.002675 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk" Oct 29 22:38:30.003063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:30.002686 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c652bcb1cb96204c005f0c22ea00db120248d0b64b8f2782b29bbbfa7b83c165" Oct 29 22:38:31.019644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.016900 4543 generic.go:356] "Generic (PLEG): container finished" podID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerID="b44d6b6e33052af8c5fa721c17c5840e708fbb246fcef7bda5d008e0391d7a34" exitCode=0 Oct 29 22:38:31.019644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.017095 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" event={"ID":"1c96591e-dfa1-4d9b-8069-cb805f13d039","Type":"ContainerDied","Data":"b44d6b6e33052af8c5fa721c17c5840e708fbb246fcef7bda5d008e0391d7a34"} Oct 29 22:38:31.188654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.188629 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:38:31.370775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.370746 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\" (UniqueName: \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\") pod \"1c96591e-dfa1-4d9b-8069-cb805f13d039\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " Oct 29 22:38:31.371769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.371028 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\") pod \"1c96591e-dfa1-4d9b-8069-cb805f13d039\" (UID: \"1c96591e-dfa1-4d9b-8069-cb805f13d039\") " Oct 29 22:38:31.371769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.371556 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume" pod "1c96591e-dfa1-4d9b-8069-cb805f13d039" (UID: "1c96591e-dfa1-4d9b-8069-cb805f13d039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:31.371973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.371872 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c96591e-dfa1-4d9b-8069-cb805f13d039-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.374462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.374426 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62" pod "1c96591e-dfa1-4d9b-8069-cb805f13d039" (UID: "1c96591e-dfa1-4d9b-8069-cb805f13d039"). InnerVolumeSpecName "kube-api-access-2qk62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:31.473089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.473038 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qk62\" (UniqueName: \"kubernetes.io/projected/1c96591e-dfa1-4d9b-8069-cb805f13d039-kube-api-access-2qk62\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.538003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.537978 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:38:31.574010 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.573982 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574045 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574131 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574239 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574264 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.574348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574304 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574402 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574463 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.574550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574487 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574564 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.574681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574645 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\") pod \"974058c5-b257-479b-a8e9-1c37b7641c98\" (UID: \"974058c5-b257-479b-a8e9-1c37b7641c98\") " Oct 29 22:38:31.575111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574841 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.575111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574793 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:31.575111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.574931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.575656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.575634 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.575701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.575665 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.575701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.575681 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.575701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.575697 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.575816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.575711 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974058c5-b257-479b-a8e9-1c37b7641c98-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.576361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.576343 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:31.576936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.576901 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.577064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.576994 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "kube-api-access-rf5nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:31.577064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.577015 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token" pod "974058c5-b257-479b-a8e9-1c37b7641c98" (UID: "974058c5-b257-479b-a8e9-1c37b7641c98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:31.793162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.789636 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.793162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.789681 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974058c5-b257-479b-a8e9-1c37b7641c98-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.793162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.789697 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5nk\" (UniqueName: \"kubernetes.io/projected/974058c5-b257-479b-a8e9-1c37b7641c98-kube-api-access-rf5nk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.793162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:31.789710 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974058c5-b257-479b-a8e9-1c37b7641c98-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:32.027137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.027103 4543 generic.go:356] "Generic (PLEG): container finished" podID="974058c5-b257-479b-a8e9-1c37b7641c98" containerID="feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33" exitCode=0 Oct 29 22:38:32.027531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.027197 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" event={"ID":"974058c5-b257-479b-a8e9-1c37b7641c98","Type":"ContainerDied","Data":"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33"} Oct 29 22:38:32.027531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.027222 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" Oct 29 22:38:32.027531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.027283 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d" event={"ID":"974058c5-b257-479b-a8e9-1c37b7641c98","Type":"ContainerDied","Data":"fa4369e5713d3f36daf9d31c79341d303572410198c8bd22ba70445c5a8d10b2"} Oct 29 22:38:32.027531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.027314 4543 scope.go:122] "RemoveContainer" containerID="feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33" Oct 29 22:38:32.029050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.029019 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" Oct 29 22:38:32.029230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.029201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l" event={"ID":"1c96591e-dfa1-4d9b-8069-cb805f13d039","Type":"ContainerDied","Data":"4f80cc2d363e77266a06ceab9858856e106f612e182acae7eba9f3c3b5698f68"} Oct 29 22:38:32.040459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.040440 4543 scope.go:122] "RemoveContainer" containerID="feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33" Oct 29 22:38:32.040715 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:32.040693 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33\": container with ID starting with feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33 not found: ID does not exist" containerID="feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33" Oct 29 22:38:32.040760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.040722 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33"} err="failed to get container status \"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33\": rpc error: code = NotFound desc = could not find container \"feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33\": container with ID starting with feb5bfee7d75e83b898bf01b486808b63752fa55d8a5af1681f261256843db33 not found: ID does not exist" Oct 29 22:38:32.040760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.040739 4543 scope.go:122] "RemoveContainer" containerID="b44d6b6e33052af8c5fa721c17c5840e708fbb246fcef7bda5d008e0391d7a34" Oct 29 22:38:32.089893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.089865 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:38:32.096899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.096875 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0412c3-c58b9cc58-t2k9l"] Oct 29 22:38:32.129938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.129909 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:38:32.138409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.138384 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-mnw5d"] Oct 29 22:38:32.389018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.388944 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" path="/var/lib/kubelet/pods/1c96591e-dfa1-4d9b-8069-cb805f13d039/volumes" Oct 29 22:38:32.389527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:32.389510 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" path="/var/lib/kubelet/pods/974058c5-b257-479b-a8e9-1c37b7641c98/volumes" Oct 29 22:38:36.086127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.084776 4543 generic.go:356] "Generic (PLEG): container finished" podID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerID="f14da10a47ca4c19f73a05f04b76a4e4b4516e5e47d2550183c3feb2e2c1e068" exitCode=0 Oct 29 22:38:36.086127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.084946 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" event={"ID":"a9ca6692-75b0-425e-82ec-fbab68fbe5a5","Type":"ContainerDied","Data":"f14da10a47ca4c19f73a05f04b76a4e4b4516e5e47d2550183c3feb2e2c1e068"} Oct 29 22:38:36.200255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.200230 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:38:36.314149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.307141 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\" (UniqueName: \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\") pod \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " Oct 29 22:38:36.314149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.307289 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\") pod \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\" (UID: \"a9ca6692-75b0-425e-82ec-fbab68fbe5a5\") " Oct 29 22:38:36.341111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.327050 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume" pod "a9ca6692-75b0-425e-82ec-fbab68fbe5a5" (UID: "a9ca6692-75b0-425e-82ec-fbab68fbe5a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:36.391190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.387881 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z" pod "a9ca6692-75b0-425e-82ec-fbab68fbe5a5" (UID: "a9ca6692-75b0-425e-82ec-fbab68fbe5a5"). InnerVolumeSpecName "kube-api-access-4xb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:36.409725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.409688 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:36.409725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.409726 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xb4z\" (UniqueName: \"kubernetes.io/projected/a9ca6692-75b0-425e-82ec-fbab68fbe5a5-kube-api-access-4xb4z\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:36.664677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:36.664645 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:38:37.095849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:37.095820 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" Oct 29 22:38:37.095849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:37.095825 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt" event={"ID":"a9ca6692-75b0-425e-82ec-fbab68fbe5a5","Type":"ContainerDied","Data":"7cd247de83012d5806799bc7148f30cb1d73065d960cf6731c6c3645db0019fa"} Oct 29 22:38:37.096396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:37.095895 4543 scope.go:122] "RemoveContainer" containerID="f14da10a47ca4c19f73a05f04b76a4e4b4516e5e47d2550183c3feb2e2c1e068" Oct 29 22:38:37.158711 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:37.158683 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:38:37.164319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:37.164299 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-50d4af-7fd867cb99-gjhkt"] Oct 29 22:38:38.392260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:38.392226 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" path="/var/lib/kubelet/pods/a9ca6692-75b0-425e-82ec-fbab68fbe5a5/volumes" Oct 29 22:38:38.774784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:38.774751 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:38:38.775327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:38.775288 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" containerID="cri-o://70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf" gracePeriod=30 Oct 29 22:38:39.357498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.357468 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b"] Oct 29 22:38:39.358495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358473 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" Oct 29 22:38:39.358495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358493 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358509 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5abe43ab-a867-424e-9522-78453a81099c" containerName="revoke-keys" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358515 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abe43ab-a867-424e-9522-78453a81099c" containerName="revoke-keys" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358525 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358530 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358539 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" containerName="revoke-keys" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358544 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" containerName="revoke-keys" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358588 4543 cpu_manager.go:414] "containerMap: removing container" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" Oct 29 22:38:39.358678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358592 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" Oct 29 22:38:39.358940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358747 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="974058c5-b257-479b-a8e9-1c37b7641c98" containerName="istio-proxy" Oct 29 22:38:39.358940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358760 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c96591e-dfa1-4d9b-8069-cb805f13d039" containerName="payload-processing" Oct 29 22:38:39.358940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358768 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abe43ab-a867-424e-9522-78453a81099c" containerName="revoke-keys" Oct 29 22:38:39.358940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358773 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ca6692-75b0-425e-82ec-fbab68fbe5a5" containerName="payload-pre-processing" Oct 29 22:38:39.358940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.358780 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" containerName="revoke-keys" Oct 29 22:38:39.364148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.364130 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.382177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.382152 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b"] Oct 29 22:38:39.446736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.446707 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.447405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.446861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.447933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.447671 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhvn\" (UniqueName: \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.468965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.468939 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:38:39.469190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.469175 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:38:39.549204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.549181 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhvn\" (UniqueName: \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.549314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.549289 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.549369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.549343 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.549906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.549881 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.551692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.551666 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.576677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.576657 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhvn\" (UniqueName: \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:39.680478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:39.680430 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:40.182357 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:40.165242 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b"] Oct 29 22:38:41.279636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:41.279600 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" event={"ID":"d7d9bdbb-d13b-4ebb-87cc-8599385da899","Type":"ContainerStarted","Data":"ffc31e6218fd25243af220741ebaa54fe5e80a4bcbf63d0cb5feb993a330260f"} Oct 29 22:38:41.280170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:41.279658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" event={"ID":"d7d9bdbb-d13b-4ebb-87cc-8599385da899","Type":"ContainerStarted","Data":"b0879e3849e9e3a1a632745dd6ebf09ae443bd3f82615e64e6c126c047c3913b"} Oct 29 22:38:41.327404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:41.327355 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" podStartSLOduration=2.327342583 podStartE2EDuration="2.327342583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:41.32322687 +0000 UTC m=+3383.065157544" watchObservedRunningTime="2026-10-29 22:38:41.327342583 +0000 UTC m=+3383.069273256" Oct 29 22:38:41.665032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:41.664959 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" podUID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" containerName="payload-processing" containerID="cri-o://0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000" gracePeriod=25 Oct 29 22:38:41.979321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:41.979296 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:38:42.117639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.117604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\") pod \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " Oct 29 22:38:42.117824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.117748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\" (UniqueName: \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\") pod \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\" (UID: \"a8493832-1e6d-44e5-a54f-c95d00e6e4c3\") " Oct 29 22:38:42.118138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.118109 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume" pod "a8493832-1e6d-44e5-a54f-c95d00e6e4c3" (UID: "a8493832-1e6d-44e5-a54f-c95d00e6e4c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:42.118553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.118530 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:42.120426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.120404 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn" pod "a8493832-1e6d-44e5-a54f-c95d00e6e4c3" (UID: "a8493832-1e6d-44e5-a54f-c95d00e6e4c3"). InnerVolumeSpecName "kube-api-access-fq5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:42.219937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.219912 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5tn\" (UniqueName: \"kubernetes.io/projected/a8493832-1e6d-44e5-a54f-c95d00e6e4c3-kube-api-access-fq5tn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:42.290017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.289989 4543 generic.go:356] "Generic (PLEG): container finished" podID="d7d9bdbb-d13b-4ebb-87cc-8599385da899" containerID="ffc31e6218fd25243af220741ebaa54fe5e80a4bcbf63d0cb5feb993a330260f" exitCode=0 Oct 29 22:38:42.290400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.290067 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" event={"ID":"d7d9bdbb-d13b-4ebb-87cc-8599385da899","Type":"ContainerDied","Data":"ffc31e6218fd25243af220741ebaa54fe5e80a4bcbf63d0cb5feb993a330260f"} Oct 29 22:38:42.291665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.291646 4543 generic.go:356] "Generic (PLEG): container finished" podID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" containerID="0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000" exitCode=0 Oct 29 22:38:42.291753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.291685 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" Oct 29 22:38:42.291753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.291730 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" event={"ID":"a8493832-1e6d-44e5-a54f-c95d00e6e4c3","Type":"ContainerDied","Data":"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000"} Oct 29 22:38:42.291860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.291791 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r" event={"ID":"a8493832-1e6d-44e5-a54f-c95d00e6e4c3","Type":"ContainerDied","Data":"0dd078fcddccfe1a65348301a6793dc9570566020682226919f374567175e392"} Oct 29 22:38:42.291860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.291854 4543 scope.go:122] "RemoveContainer" containerID="0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000" Oct 29 22:38:42.304012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.303997 4543 scope.go:122] "RemoveContainer" containerID="0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000" Oct 29 22:38:42.304297 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:42.304274 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000\": container with ID starting with 0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000 not found: ID does not exist" containerID="0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000" Oct 29 22:38:42.304365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.304307 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000"} err="failed to get container status \"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000\": rpc error: code = NotFound desc = could not find container \"0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000\": container with ID starting with 0a69479526f45656d8669380fab7ccae5907ca34e4f2ba0f6724d2841a39d000 not found: ID does not exist" Oct 29 22:38:42.392643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.392616 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:38:42.392794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:42.392651 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-797e0c-c9ffb999b-zl24r"] Oct 29 22:38:43.483867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.483840 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:43.557612 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.557546 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\") pod \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " Oct 29 22:38:43.557743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.557640 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\") pod \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " Oct 29 22:38:43.557743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.557708 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\" (UniqueName: \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\") pod \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\" (UID: \"d7d9bdbb-d13b-4ebb-87cc-8599385da899\") " Oct 29 22:38:43.558094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.558051 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca" pod "d7d9bdbb-d13b-4ebb-87cc-8599385da899" (UID: "d7d9bdbb-d13b-4ebb-87cc-8599385da899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:43.558536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.558514 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:43.560046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.560021 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn" pod "d7d9bdbb-d13b-4ebb-87cc-8599385da899" (UID: "d7d9bdbb-d13b-4ebb-87cc-8599385da899"). InnerVolumeSpecName "kube-api-access-swhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:43.560315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.560295 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca" pod "d7d9bdbb-d13b-4ebb-87cc-8599385da899" (UID: "d7d9bdbb-d13b-4ebb-87cc-8599385da899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:43.660316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.660293 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhvn\" (UniqueName: \"kubernetes.io/projected/d7d9bdbb-d13b-4ebb-87cc-8599385da899-kube-api-access-swhvn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:43.660416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:43.660319 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9bdbb-d13b-4ebb-87cc-8599385da899-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.045668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.045636 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:38:44.166113 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166026 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166123 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166182 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166231 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166330 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166327 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:44.166454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166424 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166480 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166487 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:44.166727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166615 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:44.166727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166633 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:44.166727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166644 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.166727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.166684 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\") pod \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\" (UID: \"16723af5-2c7c-4e44-a2bd-8722d3f75bcd\") " Oct 29 22:38:44.167198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167173 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:44.167518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167493 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.167599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167525 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.167599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167540 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.167599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167554 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.167599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.167567 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.168748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.168728 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:44.169107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.169067 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:44.169198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.169175 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:44.169254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.169211 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq" pod "16723af5-2c7c-4e44-a2bd-8722d3f75bcd" (UID: "16723af5-2c7c-4e44-a2bd-8722d3f75bcd"). InnerVolumeSpecName "kube-api-access-pvxqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:44.268820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.268797 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.268820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.268820 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.268956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.268831 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxqq\" (UniqueName: \"kubernetes.io/projected/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-kube-api-access-pvxqq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.268956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.268840 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16723af5-2c7c-4e44-a2bd-8722d3f75bcd-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:44.313840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.313816 4543 generic.go:356] "Generic (PLEG): container finished" podID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerID="70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf" exitCode=0 Oct 29 22:38:44.313945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.313863 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" Oct 29 22:38:44.313945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.313894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerDied","Data":"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf"} Oct 29 22:38:44.314052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.313975 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv" event={"ID":"16723af5-2c7c-4e44-a2bd-8722d3f75bcd","Type":"ContainerDied","Data":"5242b95ad6f54b14ccd0a42c8ffaa63c6754c32efefdd82a938b20effc455f7d"} Oct 29 22:38:44.314052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.314004 4543 scope.go:122] "RemoveContainer" containerID="70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf" Oct 29 22:38:44.315842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.315821 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" event={"ID":"d7d9bdbb-d13b-4ebb-87cc-8599385da899","Type":"ContainerDied","Data":"b0879e3849e9e3a1a632745dd6ebf09ae443bd3f82615e64e6c126c047c3913b"} Oct 29 22:38:44.315963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.315855 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b" Oct 29 22:38:44.315963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.315864 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0879e3849e9e3a1a632745dd6ebf09ae443bd3f82615e64e6c126c047c3913b" Oct 29 22:38:44.327798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.327777 4543 scope.go:122] "RemoveContainer" containerID="afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2" Oct 29 22:38:44.338627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.338609 4543 scope.go:122] "RemoveContainer" containerID="70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf" Oct 29 22:38:44.338883 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:44.338862 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf\": container with ID starting with 70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf not found: ID does not exist" containerID="70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf" Oct 29 22:38:44.338942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.338891 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf"} err="failed to get container status \"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf\": rpc error: code = NotFound desc = could not find container \"70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf\": container with ID starting with 70fdbe7ef9db3b62268b2f4d9cbe220878d7e5e3d976a7d1300fc57f86b70daf not found: ID does not exist" Oct 29 22:38:44.338942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.338909 4543 scope.go:122] "RemoveContainer" containerID="afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2" Oct 29 22:38:44.339244 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:44.339223 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2\": container with ID starting with afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2 not found: ID does not exist" containerID="afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2" Oct 29 22:38:44.339314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.339250 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2"} err="failed to get container status \"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2\": rpc error: code = NotFound desc = could not find container \"afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2\": container with ID starting with afb29e786ee81b6872d7eb78758f765c90e4fc021f69cab246a6cedf9105e5e2 not found: ID does not exist" Oct 29 22:38:44.388187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.388162 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" path="/var/lib/kubelet/pods/a8493832-1e6d-44e5-a54f-c95d00e6e4c3/volumes" Oct 29 22:38:44.393922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.393900 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:38:44.399052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:44.399030 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0412c3-openshift-default-64bb8d8cdf-xfxmv"] Oct 29 22:38:46.387841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:46.387807 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" path="/var/lib/kubelet/pods/16723af5-2c7c-4e44-a2bd-8722d3f75bcd/volumes" Oct 29 22:38:49.443319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.443291 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s"] Oct 29 22:38:49.444265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444247 4543 cpu_manager.go:414] "containerMap: removing container" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:38:49.444316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444266 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:38:49.444316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444285 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d7d9bdbb-d13b-4ebb-87cc-8599385da899" containerName="revoke-keys" Oct 29 22:38:49.444316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444290 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d9bdbb-d13b-4ebb-87cc-8599385da899" containerName="revoke-keys" Oct 29 22:38:49.444316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444302 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" containerName="payload-processing" Oct 29 22:38:49.444316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444306 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" containerName="payload-processing" Oct 29 22:38:49.444487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444338 4543 cpu_manager.go:414] "containerMap: removing container" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:38:49.444487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444343 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:38:49.444554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444514 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:38:49.444554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444525 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d9bdbb-d13b-4ebb-87cc-8599385da899" containerName="revoke-keys" Oct 29 22:38:49.444554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.444531 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8493832-1e6d-44e5-a54f-c95d00e6e4c3" containerName="payload-processing" Oct 29 22:38:49.448641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.448621 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.466613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.466592 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s"] Oct 29 22:38:49.536614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.536592 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:38:49.536897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.536879 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:38:49.571640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.571615 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjdl\" (UniqueName: \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.571816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.571797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.571890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.571879 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.673594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.673568 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.673684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.673639 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.673751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.673738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjdl\" (UniqueName: \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.674318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.674296 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.676103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.676064 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.700197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.700145 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjdl\" (UniqueName: \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.764184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.764164 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:49.823408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.822990 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:38:49.870623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:49.870594 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:38:50.140129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:50.134997 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s"] Oct 29 22:38:50.375463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:50.375389 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" event={"ID":"36c83cd2-96ea-4b10-bfad-75948681c028","Type":"ContainerStarted","Data":"b1cc548e896ee94f93bdea545b329662004aa5dc7bbcd8d7a846f42833b34826"} Oct 29 22:38:51.385763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:51.385728 4543 generic.go:356] "Generic (PLEG): container finished" podID="36c83cd2-96ea-4b10-bfad-75948681c028" containerID="cfbc235039840ffabe473f2bc3552e6598cc1f390d3d337eb85fe5cc8fcaeed7" exitCode=0 Oct 29 22:38:51.386191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:51.385808 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" event={"ID":"36c83cd2-96ea-4b10-bfad-75948681c028","Type":"ContainerDied","Data":"cfbc235039840ffabe473f2bc3552e6598cc1f390d3d337eb85fe5cc8fcaeed7"} Oct 29 22:38:52.571473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.571440 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:52.626532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.626504 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\") pod \"36c83cd2-96ea-4b10-bfad-75948681c028\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " Oct 29 22:38:52.626694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.626577 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\") pod \"36c83cd2-96ea-4b10-bfad-75948681c028\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " Oct 29 22:38:52.626694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.626674 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\" (UniqueName: \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\") pod \"36c83cd2-96ea-4b10-bfad-75948681c028\" (UID: \"36c83cd2-96ea-4b10-bfad-75948681c028\") " Oct 29 22:38:52.626932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.626908 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca" pod "36c83cd2-96ea-4b10-bfad-75948681c028" (UID: "36c83cd2-96ea-4b10-bfad-75948681c028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:52.627379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.627352 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:52.629460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.629432 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca" pod "36c83cd2-96ea-4b10-bfad-75948681c028" (UID: "36c83cd2-96ea-4b10-bfad-75948681c028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:52.629569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.629532 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl" pod "36c83cd2-96ea-4b10-bfad-75948681c028" (UID: "36c83cd2-96ea-4b10-bfad-75948681c028"). InnerVolumeSpecName "kube-api-access-4jjdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:52.729117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.729061 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c83cd2-96ea-4b10-bfad-75948681c028-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:52.729117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:52.729115 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjdl\" (UniqueName: \"kubernetes.io/projected/36c83cd2-96ea-4b10-bfad-75948681c028-kube-api-access-4jjdl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:38:53.405762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:53.405737 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" Oct 29 22:38:53.405921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:53.405773 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s" event={"ID":"36c83cd2-96ea-4b10-bfad-75948681c028","Type":"ContainerDied","Data":"b1cc548e896ee94f93bdea545b329662004aa5dc7bbcd8d7a846f42833b34826"} Oct 29 22:38:53.405921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:53.405816 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1cc548e896ee94f93bdea545b329662004aa5dc7bbcd8d7a846f42833b34826" Oct 29 22:38:54.823802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:54.823749 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" containerID="cri-o://aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8" gracePeriod=25 Oct 29 22:38:54.871597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:38:54.871543 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" containerID="cri-o://7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a" gracePeriod=25 Oct 29 22:38:57.961640 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:38:57.961606 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:39:03.467522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:03.467473 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:03.467937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:03.467533 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:13.467458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:13.467425 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:13.467910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:13.467437 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:15.198218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.198185 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:39:15.263498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.263461 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:39:15.360844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.360753 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\" (UniqueName: \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\") pod \"87ad99d7-332f-417e-b1a1-888db568f7a6\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " Oct 29 22:39:15.361014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.360884 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\" (UniqueName: \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\") pod \"e93e4d70-8859-441f-ab72-8d9b660b02cb\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " Oct 29 22:39:15.361090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.361045 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\") pod \"e93e4d70-8859-441f-ab72-8d9b660b02cb\" (UID: \"e93e4d70-8859-441f-ab72-8d9b660b02cb\") " Oct 29 22:39:15.361151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.361140 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\") pod \"87ad99d7-332f-417e-b1a1-888db568f7a6\" (UID: \"87ad99d7-332f-417e-b1a1-888db568f7a6\") " Oct 29 22:39:15.361683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.361657 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume" pod "87ad99d7-332f-417e-b1a1-888db568f7a6" (UID: "87ad99d7-332f-417e-b1a1-888db568f7a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:15.361993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.361935 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume" pod "e93e4d70-8859-441f-ab72-8d9b660b02cb" (UID: "e93e4d70-8859-441f-ab72-8d9b660b02cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:15.362156 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.361992 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ad99d7-332f-417e-b1a1-888db568f7a6-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:15.365366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.365336 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2" pod "87ad99d7-332f-417e-b1a1-888db568f7a6" (UID: "87ad99d7-332f-417e-b1a1-888db568f7a6"). InnerVolumeSpecName "kube-api-access-mwsp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:15.365661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.365629 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q" pod "e93e4d70-8859-441f-ab72-8d9b660b02cb" (UID: "e93e4d70-8859-441f-ab72-8d9b660b02cb"). InnerVolumeSpecName "kube-api-access-vkk4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:15.488557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.485133 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkk4q\" (UniqueName: \"kubernetes.io/projected/e93e4d70-8859-441f-ab72-8d9b660b02cb-kube-api-access-vkk4q\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:15.488804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.488785 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e4d70-8859-441f-ab72-8d9b660b02cb-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:15.488901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.488891 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsp2\" (UniqueName: \"kubernetes.io/projected/87ad99d7-332f-417e-b1a1-888db568f7a6-kube-api-access-mwsp2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:15.636718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.636630 4543 generic.go:356] "Generic (PLEG): container finished" podID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerID="7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a" exitCode=0 Oct 29 22:39:15.636718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.636700 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" Oct 29 22:39:15.636961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.636715 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" event={"ID":"e93e4d70-8859-441f-ab72-8d9b660b02cb","Type":"ContainerDied","Data":"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a"} Oct 29 22:39:15.636961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.636820 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv" event={"ID":"e93e4d70-8859-441f-ab72-8d9b660b02cb","Type":"ContainerDied","Data":"8a4630c3c756f8ee76e062a2fbd44e2bfa0dfc2c2ff6bf764dcbb34240efd7e0"} Oct 29 22:39:15.636961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.636857 4543 scope.go:122] "RemoveContainer" containerID="7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a" Oct 29 22:39:15.638440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.638412 4543 generic.go:356] "Generic (PLEG): container finished" podID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerID="aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8" exitCode=0 Oct 29 22:39:15.638559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.638476 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" Oct 29 22:39:15.638559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.638512 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" event={"ID":"87ad99d7-332f-417e-b1a1-888db568f7a6","Type":"ContainerDied","Data":"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8"} Oct 29 22:39:15.638677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.638561 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk" event={"ID":"87ad99d7-332f-417e-b1a1-888db568f7a6","Type":"ContainerDied","Data":"fe61d92164da154c726741979fce4b931f226d44d14e8ca82f9271618fa64726"} Oct 29 22:39:15.652516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.652488 4543 scope.go:122] "RemoveContainer" containerID="7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a" Oct 29 22:39:15.652855 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:15.652828 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a\": container with ID starting with 7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a not found: ID does not exist" containerID="7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a" Oct 29 22:39:15.652954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.652866 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a"} err="failed to get container status \"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a\": rpc error: code = NotFound desc = could not find container \"7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a\": container with ID starting with 7bc1e9b0de938df06d5be4aad8bda78b1640e7a0cd186aefae273c722356823a not found: ID does not exist" Oct 29 22:39:15.652954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.652890 4543 scope.go:122] "RemoveContainer" containerID="aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8" Oct 29 22:39:15.663557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.663524 4543 scope.go:122] "RemoveContainer" containerID="aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8" Oct 29 22:39:15.663816 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:15.663797 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8\": container with ID starting with aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8 not found: ID does not exist" containerID="aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8" Oct 29 22:39:15.663894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.663822 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8"} err="failed to get container status \"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8\": rpc error: code = NotFound desc = could not find container \"aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8\": container with ID starting with aafc8df4d96446868e59435f00ad5eb15943547772554918ac68b8078ec80dd8 not found: ID does not exist" Oct 29 22:39:15.700321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.700294 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:39:15.712467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.712448 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e7b56d-9c99d6495-smrsv"] Oct 29 22:39:15.743385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.743362 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:39:15.748995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:15.748976 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e7b56d-7bb97785b4-qxkjk"] Oct 29 22:39:16.389360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:16.389329 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" path="/var/lib/kubelet/pods/87ad99d7-332f-417e-b1a1-888db568f7a6/volumes" Oct 29 22:39:16.389775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:16.389760 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" path="/var/lib/kubelet/pods/e93e4d70-8859-441f-ab72-8d9b660b02cb/volumes" Oct 29 22:39:27.962849 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:27.962815 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:39:31.020345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:31.020315 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh"] Oct 29 22:39:31.033232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:31.033208 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-26abe8-ncbjh"] Oct 29 22:39:31.416255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:31.416174 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:39:31.416439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:31.416370 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" containerID="cri-o://180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569" gracePeriod=30 Oct 29 22:39:32.389388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:32.389355 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abe43ab-a867-424e-9522-78453a81099c" path="/var/lib/kubelet/pods/5abe43ab-a867-424e-9522-78453a81099c/volumes" Oct 29 22:39:35.248185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:35.248143 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:39:35.248718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:35.248216 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:39:35.354998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:35.354971 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk"] Oct 29 22:39:35.367775 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:35.367742 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-797e0c-2tbfk"] Oct 29 22:39:36.391844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.391809 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de" path="/var/lib/kubelet/pods/4f0b0800-15d6-44c8-9ebe-e0ce5a4fc1de/volumes" Oct 29 22:39:36.715447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.715423 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:39:36.756179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756147 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756267 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756332 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756374 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756460 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756456 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:36.756619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756559 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756596 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756622 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.756739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756633 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.756739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.756739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756668 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.756739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756723 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\") pod \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\" (UID: \"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f\") " Oct 29 22:39:36.757032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.756926 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.757851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.757820 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.757851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.757851 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.758054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.757867 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.758054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.757881 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.758054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.757895 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.759202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.759170 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:36.759334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.759308 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:36.759334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.759325 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.759458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.759362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j" pod "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" (UID: "d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f"). InnerVolumeSpecName "kube-api-access-s8q9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:36.859277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.859244 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.859277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.859275 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.859471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.859290 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8q9j\" (UniqueName: \"kubernetes.io/projected/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-kube-api-access-s8q9j\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.859471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.859302 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.882372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.882336 4543 generic.go:356] "Generic (PLEG): container finished" podID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerID="180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569" exitCode=0 Oct 29 22:39:36.882521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.882395 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" Oct 29 22:39:36.882521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.882415 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" event={"ID":"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f","Type":"ContainerDied","Data":"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569"} Oct 29 22:39:36.882521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.882469 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv" event={"ID":"d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f","Type":"ContainerDied","Data":"e78c4e570603dabe2f907744cd2cb056acc36d41381ca0eb8fda97cb12aca125"} Oct 29 22:39:36.882521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.882492 4543 scope.go:122] "RemoveContainer" containerID="180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569" Oct 29 22:39:36.895117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.895059 4543 scope.go:122] "RemoveContainer" containerID="180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569" Oct 29 22:39:36.895345 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:36.895327 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569\": container with ID starting with 180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569 not found: ID does not exist" containerID="180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569" Oct 29 22:39:36.895401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:36.895352 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569"} err="failed to get container status \"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569\": rpc error: code = NotFound desc = could not find container \"180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569\": container with ID starting with 180db34e2256884e5b77d60e281a8628c0a17e4f35bf50302cc741596f836569 not found: ID does not exist" Oct 29 22:39:37.041694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:37.041129 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:39:37.080109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:37.074493 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-gf4mv"] Oct 29 22:39:38.391675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:38.391639 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" path="/var/lib/kubelet/pods/d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f/volumes" Oct 29 22:39:39.718645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:39.718616 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:39:39.719061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:39.718873 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" containerID="cri-o://4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e" gracePeriod=30 Oct 29 22:39:40.423333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.423301 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz"] Oct 29 22:39:40.424220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424196 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" Oct 29 22:39:40.424220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424215 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424233 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424239 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424300 4543 cpu_manager.go:414] "containerMap: removing container" podUID="36c83cd2-96ea-4b10-bfad-75948681c028" containerName="revoke-keys" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424305 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c83cd2-96ea-4b10-bfad-75948681c028" containerName="revoke-keys" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424317 4543 cpu_manager.go:414] "containerMap: removing container" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" Oct 29 22:39:40.424453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424322 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" Oct 29 22:39:40.424820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424481 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="16723af5-2c7c-4e44-a2bd-8722d3f75bcd" containerName="istio-proxy" Oct 29 22:39:40.424820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424497 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c83cd2-96ea-4b10-bfad-75948681c028" containerName="revoke-keys" Oct 29 22:39:40.424820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424509 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ad99d7-332f-417e-b1a1-888db568f7a6" containerName="payload-processing" Oct 29 22:39:40.424820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424520 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93e4d70-8859-441f-ab72-8d9b660b02cb" containerName="payload-pre-processing" Oct 29 22:39:40.424820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.424526 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2179d4a-0713-4c8c-b1cd-1068b8d9ba3f" containerName="istio-proxy" Oct 29 22:39:40.430641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.430621 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.448027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.448006 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz"] Oct 29 22:39:40.479067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.479033 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.479255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.479228 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.479441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.479420 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cqcn\" (UniqueName: \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.485539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.485514 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:39:40.485654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.485565 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:39:40.580626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.580604 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cqcn\" (UniqueName: \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.580750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.580739 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.580823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.580807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.581433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.581412 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.583043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.583024 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.606236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.606212 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cqcn\" (UniqueName: \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:40.747407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:40.747377 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:41.133515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:41.131289 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz"] Oct 29 22:39:41.918456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:41.918430 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s"] Oct 29 22:39:41.957098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:41.956719 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05687d-vmn4s"] Oct 29 22:39:42.001095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:42.000065 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" event={"ID":"494ae681-d986-4405-a41a-03e8aef47b72","Type":"ContainerStarted","Data":"8d01908378aeea88ffb8b6cbb84a2c7090ca68f234b7f9cbb4707f3c945c088e"} Oct 29 22:39:42.001095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:42.000184 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" event={"ID":"494ae681-d986-4405-a41a-03e8aef47b72","Type":"ContainerStarted","Data":"7075b4d3b8e7c93d54625718a59a8f73d0bfb5a6b268642d740b1bd2c25d6b43"} Oct 29 22:39:42.046365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:42.046319 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" podStartSLOduration=2.046304004 podStartE2EDuration="2.046304004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:42.044496312 +0000 UTC m=+3443.786426987" watchObservedRunningTime="2026-10-29 22:39:42.046304004 +0000 UTC m=+3443.788234677" Oct 29 22:39:42.389027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:42.388990 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c83cd2-96ea-4b10-bfad-75948681c028" path="/var/lib/kubelet/pods/36c83cd2-96ea-4b10-bfad-75948681c028/volumes" Oct 29 22:39:43.010489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:43.010456 4543 generic.go:356] "Generic (PLEG): container finished" podID="494ae681-d986-4405-a41a-03e8aef47b72" containerID="8d01908378aeea88ffb8b6cbb84a2c7090ca68f234b7f9cbb4707f3c945c088e" exitCode=0 Oct 29 22:39:43.010901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:43.010541 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" event={"ID":"494ae681-d986-4405-a41a-03e8aef47b72","Type":"ContainerDied","Data":"8d01908378aeea88ffb8b6cbb84a2c7090ca68f234b7f9cbb4707f3c945c088e"} Oct 29 22:39:44.020984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.020950 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:39:44.021562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.021220 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" containerID="cri-o://a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5" gracePeriod=30 Oct 29 22:39:44.444360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.444330 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:44.612066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.612035 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\") pod \"494ae681-d986-4405-a41a-03e8aef47b72\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " Oct 29 22:39:44.612254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.612157 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\") pod \"494ae681-d986-4405-a41a-03e8aef47b72\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " Oct 29 22:39:44.612300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.612265 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\" (UniqueName: \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\") pod \"494ae681-d986-4405-a41a-03e8aef47b72\" (UID: \"494ae681-d986-4405-a41a-03e8aef47b72\") " Oct 29 22:39:44.612488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.612466 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca" pod "494ae681-d986-4405-a41a-03e8aef47b72" (UID: "494ae681-d986-4405-a41a-03e8aef47b72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:44.612881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.612809 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:44.614450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.614428 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca" pod "494ae681-d986-4405-a41a-03e8aef47b72" (UID: "494ae681-d986-4405-a41a-03e8aef47b72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:44.614565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.614456 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn" pod "494ae681-d986-4405-a41a-03e8aef47b72" (UID: "494ae681-d986-4405-a41a-03e8aef47b72"). InnerVolumeSpecName "kube-api-access-8cqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:44.626835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.626811 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84"] Oct 29 22:39:44.651450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.650648 4543 cpu_manager.go:414] "containerMap: removing container" podUID="494ae681-d986-4405-a41a-03e8aef47b72" containerName="revoke-keys" Oct 29 22:39:44.651450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.650678 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494ae681-d986-4405-a41a-03e8aef47b72" containerName="revoke-keys" Oct 29 22:39:44.651450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.651057 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="494ae681-d986-4405-a41a-03e8aef47b72" containerName="revoke-keys" Oct 29 22:39:44.660627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.660592 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84"] Oct 29 22:39:44.660771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.660707 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.714637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.714612 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494ae681-d986-4405-a41a-03e8aef47b72-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:44.714637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.714638 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cqcn\" (UniqueName: \"kubernetes.io/projected/494ae681-d986-4405-a41a-03e8aef47b72-kube-api-access-8cqcn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:44.816179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.816152 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.816317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.816209 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.816454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.816434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8b6\" (UniqueName: \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.919371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.918762 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.919371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.918965 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8b6\" (UniqueName: \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.919371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.919263 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.919676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.919584 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.922617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.922586 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.945899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.945847 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8b6\" (UniqueName: \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.977924 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.977898 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:44.983687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:44.983663 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:39:45.035769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.035735 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" Oct 29 22:39:45.036376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.035765 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz" event={"ID":"494ae681-d986-4405-a41a-03e8aef47b72","Type":"ContainerDied","Data":"7075b4d3b8e7c93d54625718a59a8f73d0bfb5a6b268642d740b1bd2c25d6b43"} Oct 29 22:39:45.036376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.035815 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7075b4d3b8e7c93d54625718a59a8f73d0bfb5a6b268642d740b1bd2c25d6b43" Oct 29 22:39:45.037625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.037598 4543 generic.go:356] "Generic (PLEG): container finished" podID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerID="4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e" exitCode=0 Oct 29 22:39:45.037749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.037636 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" Oct 29 22:39:45.037749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.037680 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" event={"ID":"0742b7e5-5e61-46b7-8c25-18cd3bee9d69","Type":"ContainerDied","Data":"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e"} Oct 29 22:39:45.037749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.037734 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk" event={"ID":"0742b7e5-5e61-46b7-8c25-18cd3bee9d69","Type":"ContainerDied","Data":"2f11492dd3c04d79713f37d7c944039e35fdb51fcfa84c29b986f51b4e231049"} Oct 29 22:39:45.037871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.037770 4543 scope.go:122] "RemoveContainer" containerID="4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e" Oct 29 22:39:45.078531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.078507 4543 scope.go:122] "RemoveContainer" containerID="4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e" Oct 29 22:39:45.078922 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:45.078893 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e\": container with ID starting with 4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e not found: ID does not exist" containerID="4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e" Oct 29 22:39:45.079011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.078936 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e"} err="failed to get container status \"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e\": rpc error: code = NotFound desc = could not find container \"4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e\": container with ID starting with 4dcc7dbcb3d053101e31dfcad9ac4134019e0d91cb431974e7e63728ad56ac4e not found: ID does not exist" Oct 29 22:39:45.121630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121601 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.121816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121691 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.121903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121837 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.121903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121884 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121920 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.121954 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122099 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122139 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122142 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.122201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122190 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\") pod \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\" (UID: \"0742b7e5-5e61-46b7-8c25-18cd3bee9d69\") " Oct 29 22:39:45.122607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122276 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.122607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122565 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.122743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122661 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:45.122874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.122814 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.123149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.123121 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.123256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.123152 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.123256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.123167 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.123256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.123180 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.123256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.123191 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.124535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.124500 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.125817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.125792 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "kube-api-access-sd7k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:45.125962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.125938 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:45.126664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.126639 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token" pod "0742b7e5-5e61-46b7-8c25-18cd3bee9d69" (UID: "0742b7e5-5e61-46b7-8c25-18cd3bee9d69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:45.224903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.224874 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7k9\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-kube-api-access-sd7k9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.224903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.224901 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.225116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.224918 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.225116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.224930 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0742b7e5-5e61-46b7-8c25-18cd3bee9d69-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.308246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.308216 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84"] Oct 29 22:39:45.308519 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:39:45.308499 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f0b477_d843_485f_80a0_366a747c38a7.slice/crio-4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1 WatchSource:0}: Error finding container 4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1: Status 404 returned error can't find the container with id 4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1 Oct 29 22:39:45.390640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.390606 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:39:45.395757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:45.395736 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-pd7dk"] Oct 29 22:39:46.069588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.062488 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" event={"ID":"72f0b477-d843-485f-80a0-366a747c38a7","Type":"ContainerStarted","Data":"fc254f1f7c20fdc09517965977411de456c305e7320686a343309d3b5f439ada"} Oct 29 22:39:46.069588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.062559 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" event={"ID":"72f0b477-d843-485f-80a0-366a747c38a7","Type":"ContainerStarted","Data":"4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1"} Oct 29 22:39:46.140102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.136321 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" podStartSLOduration=2.13629861 podStartE2EDuration="2.13629861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:46.132097893 +0000 UTC m=+3447.874028565" watchObservedRunningTime="2026-10-29 22:39:46.13629861 +0000 UTC m=+3447.878229285" Oct 29 22:39:46.429722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.429645 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" path="/var/lib/kubelet/pods/0742b7e5-5e61-46b7-8c25-18cd3bee9d69/volumes" Oct 29 22:39:46.920299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.920261 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b"] Oct 29 22:39:46.925251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:46.925227 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e7b56d-sml6b"] Oct 29 22:39:47.080917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:47.080875 4543 generic.go:356] "Generic (PLEG): container finished" podID="72f0b477-d843-485f-80a0-366a747c38a7" containerID="fc254f1f7c20fdc09517965977411de456c305e7320686a343309d3b5f439ada" exitCode=0 Oct 29 22:39:47.081433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:47.080963 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" event={"ID":"72f0b477-d843-485f-80a0-366a747c38a7","Type":"ContainerDied","Data":"fc254f1f7c20fdc09517965977411de456c305e7320686a343309d3b5f439ada"} Oct 29 22:39:48.280752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.280724 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:48.390914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.390845 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d9bdbb-d13b-4ebb-87cc-8599385da899" path="/var/lib/kubelet/pods/d7d9bdbb-d13b-4ebb-87cc-8599385da899/volumes" Oct 29 22:39:48.478972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.478943 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\") pod \"72f0b477-d843-485f-80a0-366a747c38a7\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " Oct 29 22:39:48.479181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.478996 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\") pod \"72f0b477-d843-485f-80a0-366a747c38a7\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " Oct 29 22:39:48.479181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.479115 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\" (UniqueName: \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\") pod \"72f0b477-d843-485f-80a0-366a747c38a7\" (UID: \"72f0b477-d843-485f-80a0-366a747c38a7\") " Oct 29 22:39:48.479387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.479365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca" pod "72f0b477-d843-485f-80a0-366a747c38a7" (UID: "72f0b477-d843-485f-80a0-366a747c38a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:48.479700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.479680 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:48.481286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.481267 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca" pod "72f0b477-d843-485f-80a0-366a747c38a7" (UID: "72f0b477-d843-485f-80a0-366a747c38a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:48.481437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.481415 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6" pod "72f0b477-d843-485f-80a0-366a747c38a7" (UID: "72f0b477-d843-485f-80a0-366a747c38a7"). InnerVolumeSpecName "kube-api-access-zk8b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:48.581377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.581353 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f0b477-d843-485f-80a0-366a747c38a7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:48.581508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:48.581381 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8b6\" (UniqueName: \"kubernetes.io/projected/72f0b477-d843-485f-80a0-366a747c38a7-kube-api-access-zk8b6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.102634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.102608 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" Oct 29 22:39:49.102634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.102620 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84" event={"ID":"72f0b477-d843-485f-80a0-366a747c38a7","Type":"ContainerDied","Data":"4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1"} Oct 29 22:39:49.102788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.102699 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5a9096b8e5a6e2058ad3dd6d8fcdc3817e14bd47972f38eeb8d368be0a1eb1" Oct 29 22:39:49.312289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.312259 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:39:49.493776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.493747 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.493967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.493785 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.493967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.493873 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.493967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.493929 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494128 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494163 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494216 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494238 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:49.494383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494248 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\") pod \"c5a89316-4d07-4f5a-a551-0c78ede8199d\" (UID: \"c5a89316-4d07-4f5a-a551-0c78ede8199d\") " Oct 29 22:39:49.494554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494377 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:49.494554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494471 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:49.494676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494594 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:49.494896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494799 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:49.494989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494950 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.494989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494982 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.495138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.494996 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.495138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.495011 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a89316-4d07-4f5a-a551-0c78ede8199d-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.495138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.495024 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.496794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.496771 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:49.496909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.496881 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "kube-api-access-8mtrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:49.496979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.496920 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:49.497017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.496978 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token" pod "c5a89316-4d07-4f5a-a551-0c78ede8199d" (UID: "c5a89316-4d07-4f5a-a551-0c78ede8199d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:49.596553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.596492 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.596553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.596518 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.596553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.596528 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtrr\" (UniqueName: \"kubernetes.io/projected/c5a89316-4d07-4f5a-a551-0c78ede8199d-kube-api-access-8mtrr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:49.596553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:49.596538 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a89316-4d07-4f5a-a551-0c78ede8199d-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:50.141099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.138639 4543 generic.go:356] "Generic (PLEG): container finished" podID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerID="a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5" exitCode=0 Oct 29 22:39:50.141099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.138859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerDied","Data":"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5"} Oct 29 22:39:50.141099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.138908 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" event={"ID":"c5a89316-4d07-4f5a-a551-0c78ede8199d","Type":"ContainerDied","Data":"71ab9058cf008958100204339dd7da430a8e0875c9a0af297a0312590813ba71"} Oct 29 22:39:50.141099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.138939 4543 scope.go:122] "RemoveContainer" containerID="a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5" Oct 29 22:39:50.141099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.139103 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl" Oct 29 22:39:50.201530 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:50.201497 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a89316_4d07_4f5a_a551_0c78ede8199d.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:39:50.237599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.236969 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:39:50.249410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.249382 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-797e0c-openshift-default-5df6b57465-msmvl"] Oct 29 22:39:50.250633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.250239 4543 scope.go:122] "RemoveContainer" containerID="855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d" Oct 29 22:39:50.273129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.272795 4543 scope.go:122] "RemoveContainer" containerID="a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5" Oct 29 22:39:50.273587 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:50.273439 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5\": container with ID starting with a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5 not found: ID does not exist" containerID="a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5" Oct 29 22:39:50.273587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.273481 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5"} err="failed to get container status \"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5\": rpc error: code = NotFound desc = could not find container \"a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5\": container with ID starting with a5a10908b673629a0ff1d01693004d7bc238e2261db36832d9446833ad51f6d5 not found: ID does not exist" Oct 29 22:39:50.273587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.273508 4543 scope.go:122] "RemoveContainer" containerID="855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d" Oct 29 22:39:50.274687 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:50.274630 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d\": container with ID starting with 855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d not found: ID does not exist" containerID="855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d" Oct 29 22:39:50.274687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.274665 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d"} err="failed to get container status \"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d\": rpc error: code = NotFound desc = could not find container \"855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d\": container with ID starting with 855d4983fdf9e5caa4e0e8c0df4226d9e95005d12a595cc18e77d2318009b45d not found: ID does not exist" Oct 29 22:39:50.389833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.389801 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" path="/var/lib/kubelet/pods/c5a89316-4d07-4f5a-a551-0c78ede8199d/volumes" Oct 29 22:39:50.611497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.611465 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm"] Oct 29 22:39:50.612403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612387 4543 cpu_manager.go:414] "containerMap: removing container" podUID="72f0b477-d843-485f-80a0-366a747c38a7" containerName="revoke-keys" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612404 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f0b477-d843-485f-80a0-366a747c38a7" containerName="revoke-keys" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612424 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612430 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612450 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612456 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612464 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.612472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612468 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.612697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612609 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f0b477-d843-485f-80a0-366a747c38a7" containerName="revoke-keys" Oct 29 22:39:50.612697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612619 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.612697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612629 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0742b7e5-5e61-46b7-8c25-18cd3bee9d69" containerName="istio-proxy" Oct 29 22:39:50.612697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.612635 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a89316-4d07-4f5a-a551-0c78ede8199d" containerName="istio-proxy" Oct 29 22:39:50.618023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.617992 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.634763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.634742 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm"] Oct 29 22:39:50.677400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.677373 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:39:50.677550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.677461 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:39:50.724396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.724371 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.724515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.724405 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.724657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.724640 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6lk\" (UniqueName: \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.826235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.826206 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6lk\" (UniqueName: \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.826399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.826343 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.826399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.826363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.827163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.827140 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.828832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.828808 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.852271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.852249 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6lk\" (UniqueName: \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:50.934139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:50.934086 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:51.250800 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:39:51.250770 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f214b28_e44a_4bdc_bd87_2e4d07591211.slice/crio-4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f WatchSource:0}: Error finding container 4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f: Status 404 returned error can't find the container with id 4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f Oct 29 22:39:51.252283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:51.252187 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm"] Oct 29 22:39:52.231564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:52.230175 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" event={"ID":"5f214b28-e44a-4bdc-bd87-2e4d07591211","Type":"ContainerStarted","Data":"a7605fea59ac0270b2f5da948f8ec90e1967141a682eef97d0e085e2b42edaa2"} Oct 29 22:39:52.231564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:52.230321 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" event={"ID":"5f214b28-e44a-4bdc-bd87-2e4d07591211","Type":"ContainerStarted","Data":"4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f"} Oct 29 22:39:52.282329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:52.282265 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" podStartSLOduration=2.282243963 podStartE2EDuration="2.282243963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:52.277198782 +0000 UTC m=+3454.019129456" watchObservedRunningTime="2026-10-29 22:39:52.282243963 +0000 UTC m=+3454.024174637" Oct 29 22:39:53.241034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:53.240996 4543 generic.go:356] "Generic (PLEG): container finished" podID="5f214b28-e44a-4bdc-bd87-2e4d07591211" containerID="a7605fea59ac0270b2f5da948f8ec90e1967141a682eef97d0e085e2b42edaa2" exitCode=0 Oct 29 22:39:53.241446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:53.241098 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" event={"ID":"5f214b28-e44a-4bdc-bd87-2e4d07591211","Type":"ContainerDied","Data":"a7605fea59ac0270b2f5da948f8ec90e1967141a682eef97d0e085e2b42edaa2"} Oct 29 22:39:54.440023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.439999 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:54.553038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.548849 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\" (UniqueName: \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\") pod \"5f214b28-e44a-4bdc-bd87-2e4d07591211\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " Oct 29 22:39:54.555280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.549061 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\") pod \"5f214b28-e44a-4bdc-bd87-2e4d07591211\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " Oct 29 22:39:54.555973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.555780 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\") pod \"5f214b28-e44a-4bdc-bd87-2e4d07591211\" (UID: \"5f214b28-e44a-4bdc-bd87-2e4d07591211\") " Oct 29 22:39:54.557654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.555233 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca" pod "5f214b28-e44a-4bdc-bd87-2e4d07591211" (UID: "5f214b28-e44a-4bdc-bd87-2e4d07591211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:54.564176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.564126 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk" pod "5f214b28-e44a-4bdc-bd87-2e4d07591211" (UID: "5f214b28-e44a-4bdc-bd87-2e4d07591211"). InnerVolumeSpecName "kube-api-access-2p6lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:54.567472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.567436 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca" pod "5f214b28-e44a-4bdc-bd87-2e4d07591211" (UID: "5f214b28-e44a-4bdc-bd87-2e4d07591211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:54.658860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.658824 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p6lk\" (UniqueName: \"kubernetes.io/projected/5f214b28-e44a-4bdc-bd87-2e4d07591211-kube-api-access-2p6lk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:54.658860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.658857 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:54.659017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.658871 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f214b28-e44a-4bdc-bd87-2e4d07591211-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:54.957272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.957198 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:39:54.957497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:54.957465 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" containerID="cri-o://d8b58decc33b5ad48745396e0bd4189f1dcbbdfb2e801e5b7826c483bd82f368" gracePeriod=30 Oct 29 22:39:55.264003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.263964 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" Oct 29 22:39:55.264246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.264003 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm" event={"ID":"5f214b28-e44a-4bdc-bd87-2e4d07591211","Type":"ContainerDied","Data":"4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f"} Oct 29 22:39:55.264246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.264050 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6fc597422757ba8e5c3542edc7bb0696624fb8c3bbdf2a28be8cf7f363254f" Oct 29 22:39:55.544435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.544351 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl"] Oct 29 22:39:55.545866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.545836 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5f214b28-e44a-4bdc-bd87-2e4d07591211" containerName="revoke-keys" Oct 29 22:39:55.545866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.545865 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f214b28-e44a-4bdc-bd87-2e4d07591211" containerName="revoke-keys" Oct 29 22:39:55.546141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.546123 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f214b28-e44a-4bdc-bd87-2e4d07591211" containerName="revoke-keys" Oct 29 22:39:55.551004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.550989 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.566132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.566111 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl"] Oct 29 22:39:55.613839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.613800 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.614115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.614095 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.614254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.614238 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rv9\" (UniqueName: \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.622088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.622051 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:39:55.622246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.622230 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:39:55.716136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.716110 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.716241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.716148 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.716241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.716171 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rv9\" (UniqueName: \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.716846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.716811 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.718653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.718630 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.741815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.741791 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rv9\" (UniqueName: \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:55.868288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:55.868241 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:56.038027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:56.037995 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq"] Oct 29 22:39:56.051821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:56.051786 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c86ab4-xznpq"] Oct 29 22:39:56.179849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:56.179821 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl"] Oct 29 22:39:56.180930 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:39:56.180898 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bca4eb5_ccbb_4cea_bbd5_3212ff69dd64.slice/crio-52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99 WatchSource:0}: Error finding container 52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99: Status 404 returned error can't find the container with id 52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99 Oct 29 22:39:56.273667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:56.273639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" event={"ID":"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64","Type":"ContainerStarted","Data":"52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99"} Oct 29 22:39:56.388444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:56.388417 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f45e13-d301-4a71-ba7f-d4212992222a" path="/var/lib/kubelet/pods/e3f45e13-d301-4a71-ba7f-d4212992222a/volumes" Oct 29 22:39:57.325100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:57.324935 4543 generic.go:356] "Generic (PLEG): container finished" podID="2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" containerID="41a095872413f1856c748c05e4c0c14757ab397a2e674fd111eb045bf95ee8a4" exitCode=0 Oct 29 22:39:57.325100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:57.325047 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" event={"ID":"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64","Type":"ContainerDied","Data":"41a095872413f1856c748c05e4c0c14757ab397a2e674fd111eb045bf95ee8a4"} Oct 29 22:39:57.961983 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:39:57.961951 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:39:58.509276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.509250 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:58.580719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.580687 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\") pod \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " Oct 29 22:39:58.580926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.580899 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\" (UniqueName: \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\") pod \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " Oct 29 22:39:58.581127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.581109 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\") pod \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\" (UID: \"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64\") " Oct 29 22:39:58.581191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.581161 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca" pod "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" (UID: "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:58.581946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.581919 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:58.583340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.583312 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9" pod "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" (UID: "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64"). InnerVolumeSpecName "kube-api-access-65rv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:58.583789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.583768 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca" pod "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" (UID: "2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:58.683720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.683675 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rv9\" (UniqueName: \"kubernetes.io/projected/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-kube-api-access-65rv9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:58.683720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:58.683718 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:39:59.343557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:59.343487 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" Oct 29 22:39:59.343557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:59.343504 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl" event={"ID":"2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64","Type":"ContainerDied","Data":"52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99"} Oct 29 22:39:59.343557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:39:59.343552 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52044905fc7b655510c4aadd56608e317862ab073cb91c656a79a17d483a5e99" Oct 29 22:40:00.484989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.484940 4543 generic.go:356] "Generic (PLEG): container finished" podID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerID="d8b58decc33b5ad48745396e0bd4189f1dcbbdfb2e801e5b7826c483bd82f368" exitCode=0 Oct 29 22:40:00.485566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.485216 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerDied","Data":"d8b58decc33b5ad48745396e0bd4189f1dcbbdfb2e801e5b7826c483bd82f368"} Oct 29 22:40:00.485566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.485276 4543 scope.go:122] "RemoveContainer" containerID="7be5bba1238928e4b663decfda0fb4698bae106748324f61f56936ef1e1ae11e" Oct 29 22:40:00.528444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.528420 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:40:00.665267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665226 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665279 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665367 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665434 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665465 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665490 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:00.665663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665527 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665597 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665641 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:00.665940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665663 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665725 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\") pod \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\" (UID: \"89f7ed30-5a2e-417d-9c60-cdf53104ef89\") " Oct 29 22:40:00.665940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.665843 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:00.666174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666012 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:00.666394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:00.666805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666780 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.666891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666812 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.666891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666828 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.666891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666841 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.666891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.666857 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.667886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.667858 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:00.667994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.667901 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:00.667994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.667956 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:00.668119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.668021 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9" pod "89f7ed30-5a2e-417d-9c60-cdf53104ef89" (UID: "89f7ed30-5a2e-417d-9c60-cdf53104ef89"). InnerVolumeSpecName "kube-api-access-kszh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:00.768223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.768175 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kszh9\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-kube-api-access-kszh9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.768223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.768196 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.768223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.768206 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.768223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.768214 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7ed30-5a2e-417d-9c60-cdf53104ef89-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:00.913896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:00.913865 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:40:01.495522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:01.495487 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" event={"ID":"89f7ed30-5a2e-417d-9c60-cdf53104ef89","Type":"ContainerDied","Data":"dcd613599788bda0c1dc4d680498fe3b585c48f7c78ed7f1e7e04e439636209d"} Oct 29 22:40:01.495522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:01.495511 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br" Oct 29 22:40:01.496133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:01.495542 4543 scope.go:122] "RemoveContainer" containerID="d8b58decc33b5ad48745396e0bd4189f1dcbbdfb2e801e5b7826c483bd82f368" Oct 29 22:40:01.556621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:01.556599 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:40:01.569681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:01.569656 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e7b56d-openshift-default-7b5f79c769-xn8br"] Oct 29 22:40:02.388470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:02.388433 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" path="/var/lib/kubelet/pods/89f7ed30-5a2e-417d-9c60-cdf53104ef89/volumes" Oct 29 22:40:05.916401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:05.916356 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" containerID="cri-o://a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26" gracePeriod=25 Oct 29 22:40:06.050462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.050400 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9"] Oct 29 22:40:06.060658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.060621 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8348a7a-ts5r9"] Oct 29 22:40:06.171854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.171773 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:40:06.202993 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.202963 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:40:06.367823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.367791 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm"] Oct 29 22:40:06.373317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.373292 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-76c1b7-wfdhm"] Oct 29 22:40:06.389487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.389457 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ec18a8-ff8c-484c-930c-9ceae476480f" path="/var/lib/kubelet/pods/10ec18a8-ff8c-484c-930c-9ceae476480f/volumes" Oct 29 22:40:06.389985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:06.389969 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f214b28-e44a-4bdc-bd87-2e4d07591211" path="/var/lib/kubelet/pods/5f214b28-e44a-4bdc-bd87-2e4d07591211/volumes" Oct 29 22:40:09.057392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:09.057363 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz"] Oct 29 22:40:09.073222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:09.073193 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fe78e-4f9nz"] Oct 29 22:40:10.390511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:10.390476 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494ae681-d986-4405-a41a-03e8aef47b72" path="/var/lib/kubelet/pods/494ae681-d986-4405-a41a-03e8aef47b72/volumes" Oct 29 22:40:11.176246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.176145 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" podUID="030e8810-2efb-49bf-a641-2bd99adc75e4" containerName="payload-processing" containerID="cri-o://ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b" gracePeriod=25 Oct 29 22:40:11.206778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.203386 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" podUID="fca88882-9d81-4711-8c2f-320d1ada85f4" containerName="payload-pre-processing" containerID="cri-o://d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a" gracePeriod=25 Oct 29 22:40:11.587912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.587879 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:40:11.626310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.626287 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:40:11.669170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.669133 4543 generic.go:356] "Generic (PLEG): container finished" podID="030e8810-2efb-49bf-a641-2bd99adc75e4" containerID="ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b" exitCode=0 Oct 29 22:40:11.669363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.669191 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" Oct 29 22:40:11.669483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.669445 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" event={"ID":"030e8810-2efb-49bf-a641-2bd99adc75e4","Type":"ContainerDied","Data":"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b"} Oct 29 22:40:11.669578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.669502 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm" event={"ID":"030e8810-2efb-49bf-a641-2bd99adc75e4","Type":"ContainerDied","Data":"08962bb8998403fb03e9d64e06f90210464dae0bd9ddbc6649646da55304e7c1"} Oct 29 22:40:11.669578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.669534 4543 scope.go:122] "RemoveContainer" containerID="ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b" Oct 29 22:40:11.672018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.671995 4543 generic.go:356] "Generic (PLEG): container finished" podID="fca88882-9d81-4711-8c2f-320d1ada85f4" containerID="d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a" exitCode=0 Oct 29 22:40:11.672215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.672198 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" event={"ID":"fca88882-9d81-4711-8c2f-320d1ada85f4","Type":"ContainerDied","Data":"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a"} Oct 29 22:40:11.672273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.672244 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" event={"ID":"fca88882-9d81-4711-8c2f-320d1ada85f4","Type":"ContainerDied","Data":"c4113b436cee4a8381d6a8cd404aa13b91c20f198941a36d705ec0203196cd7b"} Oct 29 22:40:11.672326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.672315 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm" Oct 29 22:40:11.689426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.689405 4543 scope.go:122] "RemoveContainer" containerID="ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b" Oct 29 22:40:11.690314 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:11.690176 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b\": container with ID starting with ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b not found: ID does not exist" containerID="ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b" Oct 29 22:40:11.690314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.690213 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b"} err="failed to get container status \"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b\": rpc error: code = NotFound desc = could not find container \"ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b\": container with ID starting with ff0eccbdb58dffbb5a2055f144328e06ef7054ab57363d4bd24ff78fb7aa7f4b not found: ID does not exist" Oct 29 22:40:11.690314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.690237 4543 scope.go:122] "RemoveContainer" containerID="d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a" Oct 29 22:40:11.692231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.692213 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\" (UniqueName: \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\") pod \"030e8810-2efb-49bf-a641-2bd99adc75e4\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " Oct 29 22:40:11.692337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.692288 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\" (UniqueName: \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\") pod \"fca88882-9d81-4711-8c2f-320d1ada85f4\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " Oct 29 22:40:11.692337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.692322 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\") pod \"fca88882-9d81-4711-8c2f-320d1ada85f4\" (UID: \"fca88882-9d81-4711-8c2f-320d1ada85f4\") " Oct 29 22:40:11.692464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.692385 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\") pod \"030e8810-2efb-49bf-a641-2bd99adc75e4\" (UID: \"030e8810-2efb-49bf-a641-2bd99adc75e4\") " Oct 29 22:40:11.693155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.692778 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume" pod "fca88882-9d81-4711-8c2f-320d1ada85f4" (UID: "fca88882-9d81-4711-8c2f-320d1ada85f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:11.693155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.693116 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume" pod "030e8810-2efb-49bf-a641-2bd99adc75e4" (UID: "030e8810-2efb-49bf-a641-2bd99adc75e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:11.694204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.693572 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca88882-9d81-4711-8c2f-320d1ada85f4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:11.694204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.693603 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e8810-2efb-49bf-a641-2bd99adc75e4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:11.695474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.695445 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv" pod "fca88882-9d81-4711-8c2f-320d1ada85f4" (UID: "fca88882-9d81-4711-8c2f-320d1ada85f4"). InnerVolumeSpecName "kube-api-access-bwsbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:11.703433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.703407 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z" pod "030e8810-2efb-49bf-a641-2bd99adc75e4" (UID: "030e8810-2efb-49bf-a641-2bd99adc75e4"). InnerVolumeSpecName "kube-api-access-pdd5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:11.707476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.707454 4543 scope.go:122] "RemoveContainer" containerID="d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a" Oct 29 22:40:11.707935 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:11.707908 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a\": container with ID starting with d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a not found: ID does not exist" containerID="d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a" Oct 29 22:40:11.708034 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.707939 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a"} err="failed to get container status \"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a\": rpc error: code = NotFound desc = could not find container \"d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a\": container with ID starting with d605c0639725ac0c78206190df514b93d0ae060bd18f60dbffeee323b4bc203a not found: ID does not exist" Oct 29 22:40:11.795310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.795283 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdd5z\" (UniqueName: \"kubernetes.io/projected/030e8810-2efb-49bf-a641-2bd99adc75e4-kube-api-access-pdd5z\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:11.795435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:11.795313 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwsbv\" (UniqueName: \"kubernetes.io/projected/fca88882-9d81-4711-8c2f-320d1ada85f4-kube-api-access-bwsbv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.034618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.034587 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:40:12.044817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.044794 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0a3d53-57bb9bc666-46xrm"] Oct 29 22:40:12.083464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.083406 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:40:12.089463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.089436 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0a3d53-59b945cf9f-8h8gm"] Oct 29 22:40:12.389642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.389563 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030e8810-2efb-49bf-a641-2bd99adc75e4" path="/var/lib/kubelet/pods/030e8810-2efb-49bf-a641-2bd99adc75e4/volumes" Oct 29 22:40:12.390036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.390015 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca88882-9d81-4711-8c2f-320d1ada85f4" path="/var/lib/kubelet/pods/fca88882-9d81-4711-8c2f-320d1ada85f4/volumes" Oct 29 22:40:12.684279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.684222 4543 generic.go:356] "Generic (PLEG): container finished" podID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerID="fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83" exitCode=0 Oct 29 22:40:12.684646 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.684289 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerDied","Data":"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83"} Oct 29 22:40:12.684751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.684735 4543 scope.go:122] "RemoveContainer" containerID="fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83" Oct 29 22:40:12.686716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.686683 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:12.686847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.686775 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:12.686847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:12.686825 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:40:13.695696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:13.695658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerStarted","Data":"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085"} Oct 29 22:40:14.207499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.207457 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:40:14.463731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.463518 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:40:14.465020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.464987 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:14.465312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.465267 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:14.475842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.475820 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:40:14.476538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:14.476218 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" containerID="cri-o://55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435" gracePeriod=30 Oct 29 22:40:15.256430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.256389 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj"] Oct 29 22:40:15.257385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257359 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" containerName="revoke-keys" Oct 29 22:40:15.257385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257384 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" containerName="revoke-keys" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257406 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fca88882-9d81-4711-8c2f-320d1ada85f4" containerName="payload-pre-processing" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257415 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca88882-9d81-4711-8c2f-320d1ada85f4" containerName="payload-pre-processing" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257428 4543 cpu_manager.go:414] "containerMap: removing container" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257435 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257452 4543 cpu_manager.go:414] "containerMap: removing container" podUID="030e8810-2efb-49bf-a641-2bd99adc75e4" containerName="payload-processing" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257459 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030e8810-2efb-49bf-a641-2bd99adc75e4" containerName="payload-processing" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257515 4543 cpu_manager.go:414] "containerMap: removing container" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.257610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257524 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.258184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257754 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" containerName="revoke-keys" Oct 29 22:40:15.258184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257784 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca88882-9d81-4711-8c2f-320d1ada85f4" containerName="payload-pre-processing" Oct 29 22:40:15.258184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257799 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.258184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257810 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="030e8810-2efb-49bf-a641-2bd99adc75e4" containerName="payload-processing" Oct 29 22:40:15.258184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.257824 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f7ed30-5a2e-417d-9c60-cdf53104ef89" containerName="istio-proxy" Oct 29 22:40:15.260876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.260858 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.273350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.273196 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj"] Oct 29 22:40:15.353100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.349355 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:40:15.353100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.349423 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:40:15.378465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.378436 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.378598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.378481 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.378644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.378603 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qw5t\" (UniqueName: \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.451099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.450482 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:15.451099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.450536 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:15.480254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.480226 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.480411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.480279 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.480479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.480421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw5t\" (UniqueName: \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.481164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.481090 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.482845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.482820 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.506300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.506274 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw5t\" (UniqueName: \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\") pod \"maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.579344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.579269 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:15.659046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.657829 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:15.921410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:15.921372 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj"] Oct 29 22:40:15.922641 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:40:15.922609 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d528022_6d87_45e8_a500_d60c9e40042a.slice/crio-72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c WatchSource:0}: Error finding container 72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c: Status 404 returned error can't find the container with id 72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c Oct 29 22:40:16.454257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:16.454202 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:16.454762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:16.454260 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:16.784580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:16.784542 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" event={"ID":"5d528022-6d87-45e8-a500-d60c9e40042a","Type":"ContainerStarted","Data":"41961e1cd74505da4b412ff2e2dc3e98b830f644e14cdaa568aa5e68ce30f143"} Oct 29 22:40:16.784736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:16.784598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" event={"ID":"5d528022-6d87-45e8-a500-d60c9e40042a","Type":"ContainerStarted","Data":"72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c"} Oct 29 22:40:16.841629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:16.841570 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" podStartSLOduration=1.841521294 podStartE2EDuration="1.841521294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:16.828287698 +0000 UTC m=+3478.570218373" watchObservedRunningTime="2026-10-29 22:40:16.841521294 +0000 UTC m=+3478.583451968" Oct 29 22:40:17.457235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:17.454417 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:17.457235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:17.454480 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:17.795036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:17.794999 4543 generic.go:356] "Generic (PLEG): container finished" podID="5d528022-6d87-45e8-a500-d60c9e40042a" containerID="41961e1cd74505da4b412ff2e2dc3e98b830f644e14cdaa568aa5e68ce30f143" exitCode=0 Oct 29 22:40:17.795256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:17.795097 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" event={"ID":"5d528022-6d87-45e8-a500-d60c9e40042a","Type":"ContainerDied","Data":"41961e1cd74505da4b412ff2e2dc3e98b830f644e14cdaa568aa5e68ce30f143"} Oct 29 22:40:18.454468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.454434 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:18.454675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.454493 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:18.476200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.476170 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz"] Oct 29 22:40:18.480400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.480383 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.500634 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.500613 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz"] Oct 29 22:40:18.589347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.589317 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.589565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.589537 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.589648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.589588 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqgk\" (UniqueName: \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.702102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.701340 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.702102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.701413 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqgk\" (UniqueName: \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.702102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.701611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.711303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.703839 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.722404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.719661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.750107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.743101 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqgk\" (UniqueName: \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\") pod \"maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:18.794932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:18.794910 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:19.088266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.088241 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:19.163957 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:40:19.163907 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode875ebaa_e8ec_46e6_8a38_940676474d71.slice/crio-f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3 WatchSource:0}: Error finding container f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3: Status 404 returned error can't find the container with id f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3 Oct 29 22:40:19.174620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.174595 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz"] Oct 29 22:40:19.214729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.214664 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\" (UniqueName: \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\") pod \"5d528022-6d87-45e8-a500-d60c9e40042a\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " Oct 29 22:40:19.214851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.214802 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\") pod \"5d528022-6d87-45e8-a500-d60c9e40042a\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " Oct 29 22:40:19.215029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.215013 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\") pod \"5d528022-6d87-45e8-a500-d60c9e40042a\" (UID: \"5d528022-6d87-45e8-a500-d60c9e40042a\") " Oct 29 22:40:19.215384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.215345 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca" pod "5d528022-6d87-45e8-a500-d60c9e40042a" (UID: "5d528022-6d87-45e8-a500-d60c9e40042a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:19.215738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.215717 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.216891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.216872 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t" pod "5d528022-6d87-45e8-a500-d60c9e40042a" (UID: "5d528022-6d87-45e8-a500-d60c9e40042a"). InnerVolumeSpecName "kube-api-access-9qw5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:19.216961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.216879 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca" pod "5d528022-6d87-45e8-a500-d60c9e40042a" (UID: "5d528022-6d87-45e8-a500-d60c9e40042a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:19.317375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.317334 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qw5t\" (UniqueName: \"kubernetes.io/projected/5d528022-6d87-45e8-a500-d60c9e40042a-kube-api-access-9qw5t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.317375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.317372 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d528022-6d87-45e8-a500-d60c9e40042a-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.451102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.448422 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" start-of-body= Oct 29 22:40:19.451102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.448479 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.163:15021/healthz/ready\": dial tcp 10.133.1.163:15021: connect: connection refused" Oct 29 22:40:19.817050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.817020 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:40:19.826125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.825741 4543 generic.go:356] "Generic (PLEG): container finished" podID="db05227c-5018-4052-a277-4864230a5a65" containerID="55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435" exitCode=0 Oct 29 22:40:19.826125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.825949 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" Oct 29 22:40:19.826125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.826117 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerDied","Data":"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435"} Oct 29 22:40:19.826391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.826203 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz" event={"ID":"db05227c-5018-4052-a277-4864230a5a65","Type":"ContainerDied","Data":"15e1be0cd158d8836e6850d52e73f0a1e9aed03cfb0329412efdc60f2b2ebd86"} Oct 29 22:40:19.826391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.826237 4543 scope.go:122] "RemoveContainer" containerID="55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435" Oct 29 22:40:19.828728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.828695 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" event={"ID":"5d528022-6d87-45e8-a500-d60c9e40042a","Type":"ContainerDied","Data":"72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c"} Oct 29 22:40:19.828839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.828745 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72091cce72e3975f6f72ec1a8b37f71fe920a57cb28ab858523b2385fc71f86c" Oct 29 22:40:19.828839 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.828806 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj" Oct 29 22:40:19.834665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.834636 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" event={"ID":"e875ebaa-e8ec-46e6-8a38-940676474d71","Type":"ContainerStarted","Data":"67e07e266c30aaef4989066ff9737869385502be9717c212637dca84ba121609"} Oct 29 22:40:19.834797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.834695 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" event={"ID":"e875ebaa-e8ec-46e6-8a38-940676474d71","Type":"ContainerStarted","Data":"f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3"} Oct 29 22:40:19.841437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.841422 4543 scope.go:122] "RemoveContainer" containerID="1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8" Oct 29 22:40:19.853057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.853034 4543 scope.go:122] "RemoveContainer" containerID="55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435" Oct 29 22:40:19.853426 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:19.853402 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435\": container with ID starting with 55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435 not found: ID does not exist" containerID="55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435" Oct 29 22:40:19.853547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.853435 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435"} err="failed to get container status \"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435\": rpc error: code = NotFound desc = could not find container \"55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435\": container with ID starting with 55eb369096cb6d13f5e080b8b764b405607684d5cde6f292b1cedcae0f4c7435 not found: ID does not exist" Oct 29 22:40:19.853547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.853457 4543 scope.go:122] "RemoveContainer" containerID="1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8" Oct 29 22:40:19.853822 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:19.853802 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8\": container with ID starting with 1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8 not found: ID does not exist" containerID="1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8" Oct 29 22:40:19.853899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.853832 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8"} err="failed to get container status \"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8\": rpc error: code = NotFound desc = could not find container \"1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8\": container with ID starting with 1e1f99dd50ed2cc9c95f2d82ddee73f0b8386def44699ef189204293c2362de8 not found: ID does not exist" Oct 29 22:40:19.920583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.920527 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" podStartSLOduration=1.9205114 podStartE2EDuration="1.9205114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:19.914665824 +0000 UTC m=+3481.656596495" watchObservedRunningTime="2026-10-29 22:40:19.9205114 +0000 UTC m=+3481.662442073" Oct 29 22:40:19.925443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925417 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925478 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925645 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925703 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925735 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925828 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.925955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925886 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925915 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.925955 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925918 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.926141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.925952 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.926141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.926046 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\") pod \"db05227c-5018-4052-a277-4864230a5a65\" (UID: \"db05227c-5018-4052-a277-4864230a5a65\") " Oct 29 22:40:19.926330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.926309 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.926528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.926447 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:19.927214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.927194 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.927310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.927217 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.927310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.927232 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.927310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.927244 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.927310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.927258 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05227c-5018-4052-a277-4864230a5a65-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.928040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.928021 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:19.928190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.928169 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:19.928579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.928559 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "kube-api-access-llgps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:19.928648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.928570 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy" pod "db05227c-5018-4052-a277-4864230a5a65" (UID: "db05227c-5018-4052-a277-4864230a5a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.981882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.981854 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 22:40:19.990756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:19.990730 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 22:40:20.028462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.028406 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgps\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-kube-api-access-llgps\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.028462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.028434 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05227c-5018-4052-a277-4864230a5a65-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.028462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.028445 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05227c-5018-4052-a277-4864230a5a65-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.028462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.028452 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05227c-5018-4052-a277-4864230a5a65-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.180237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.180114 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:40:20.184793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.184769 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-fxqrz"] Oct 29 22:40:20.388718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.388690 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db05227c-5018-4052-a277-4864230a5a65" path="/var/lib/kubelet/pods/db05227c-5018-4052-a277-4864230a5a65/volumes" Oct 29 22:40:20.460977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.460934 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:40:20.462713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.462686 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:40:20.844779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.844748 4543 generic.go:356] "Generic (PLEG): container finished" podID="e875ebaa-e8ec-46e6-8a38-940676474d71" containerID="67e07e266c30aaef4989066ff9737869385502be9717c212637dca84ba121609" exitCode=0 Oct 29 22:40:20.845242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:20.844795 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" event={"ID":"e875ebaa-e8ec-46e6-8a38-940676474d71","Type":"ContainerDied","Data":"67e07e266c30aaef4989066ff9737869385502be9717c212637dca84ba121609"} Oct 29 22:40:22.078608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.076862 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:22.170252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.170175 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\") pod \"e875ebaa-e8ec-46e6-8a38-940676474d71\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " Oct 29 22:40:22.170397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.170361 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\") pod \"e875ebaa-e8ec-46e6-8a38-940676474d71\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " Oct 29 22:40:22.170597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.170569 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca" pod "e875ebaa-e8ec-46e6-8a38-940676474d71" (UID: "e875ebaa-e8ec-46e6-8a38-940676474d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:22.171124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.171098 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.172469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.172445 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca" pod "e875ebaa-e8ec-46e6-8a38-940676474d71" (UID: "e875ebaa-e8ec-46e6-8a38-940676474d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:22.272166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.272139 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\" (UniqueName: \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\") pod \"e875ebaa-e8ec-46e6-8a38-940676474d71\" (UID: \"e875ebaa-e8ec-46e6-8a38-940676474d71\") " Oct 29 22:40:22.272593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.272578 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e875ebaa-e8ec-46e6-8a38-940676474d71-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.274254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.274231 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk" pod "e875ebaa-e8ec-46e6-8a38-940676474d71" (UID: "e875ebaa-e8ec-46e6-8a38-940676474d71"). InnerVolumeSpecName "kube-api-access-kpqgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:22.374310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.374287 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqgk\" (UniqueName: \"kubernetes.io/projected/e875ebaa-e8ec-46e6-8a38-940676474d71-kube-api-access-kpqgk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:22.865319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.865288 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" Oct 29 22:40:22.865460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.865315 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz" event={"ID":"e875ebaa-e8ec-46e6-8a38-940676474d71","Type":"ContainerDied","Data":"f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3"} Oct 29 22:40:22.865460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:22.865372 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f52d4a861665ae1852aae35bff536fd085d7fddc33aad6d9825f2cba5a6087d3" Oct 29 22:40:25.655794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:25.655757 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:25.734783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:25.734756 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 22:40:25.777586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:25.777563 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 22:40:26.281899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.281877 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:40:26.335334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.335304 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\") pod \"94f25693-735f-44db-852b-c78561e828de\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " Oct 29 22:40:26.335480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.335360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\" (UniqueName: \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\") pod \"94f25693-735f-44db-852b-c78561e828de\" (UID: \"94f25693-735f-44db-852b-c78561e828de\") " Oct 29 22:40:26.335676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.335653 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume" pod "94f25693-735f-44db-852b-c78561e828de" (UID: "94f25693-735f-44db-852b-c78561e828de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:26.336124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.336102 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f25693-735f-44db-852b-c78561e828de-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:26.337693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.337665 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647" pod "94f25693-735f-44db-852b-c78561e828de" (UID: "94f25693-735f-44db-852b-c78561e828de"). InnerVolumeSpecName "kube-api-access-c4647". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:26.452289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.444898 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4647\" (UniqueName: \"kubernetes.io/projected/94f25693-735f-44db-852b-c78561e828de-kube-api-access-c4647\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:26.904836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.904801 4543 generic.go:356] "Generic (PLEG): container finished" podID="94f25693-735f-44db-852b-c78561e828de" containerID="a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26" exitCode=0 Oct 29 22:40:26.905358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.904844 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" Oct 29 22:40:26.905358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.904879 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" event={"ID":"94f25693-735f-44db-852b-c78561e828de","Type":"ContainerDied","Data":"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26"} Oct 29 22:40:26.905358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.904948 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6" event={"ID":"94f25693-735f-44db-852b-c78561e828de","Type":"ContainerDied","Data":"809a09f77ebf8bda369b7dfc07921e60310b735383dee234b9f24f0a8126bb24"} Oct 29 22:40:26.905358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.904980 4543 scope.go:122] "RemoveContainer" containerID="a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26" Oct 29 22:40:26.917239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.917216 4543 scope.go:122] "RemoveContainer" containerID="a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26" Oct 29 22:40:26.917508 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:26.917488 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26\": container with ID starting with a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26 not found: ID does not exist" containerID="a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26" Oct 29 22:40:26.917591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.917514 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26"} err="failed to get container status \"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26\": rpc error: code = NotFound desc = could not find container \"a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26\": container with ID starting with a024a4698ebc8471f016c81a6e35c8b0e412f58c13bba022458c477ea3d52f26 not found: ID does not exist" Oct 29 22:40:26.964907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.964886 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:40:26.977627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:26.977607 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-76c1b7-849754888b-xldd6"] Oct 29 22:40:27.961886 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:27.961852 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:40:28.391728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:28.391650 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f25693-735f-44db-852b-c78561e828de" path="/var/lib/kubelet/pods/94f25693-735f-44db-852b-c78561e828de/volumes" Oct 29 22:40:30.735006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:30.734960 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" containerID="cri-o://7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9" gracePeriod=25 Oct 29 22:40:30.777818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:30.777779 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" containerID="cri-o://95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9" gracePeriod=25 Oct 29 22:40:31.973321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:31.973280 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:32.975650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:32.975614 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:35.021104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.016323 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" containerID="cri-o://78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52" gracePeriod=45 Oct 29 22:40:35.022024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.021525 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" containerID="cri-o://533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81" gracePeriod=45 Oct 29 22:40:35.089838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.089802 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Oct 29 22:40:35.345452 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.345419 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:40:35.349283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.349263 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:40:35.474863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.474825 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.474967 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475068 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475121 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.475171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475153 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.475359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475217 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475300 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.475359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475322 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.475542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475384 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.475542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475517 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.475666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475566 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\" (UniqueName: \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475641 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\" (UniqueName: \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.475666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.475848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475701 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\") pod \"987893f5-5826-4d00-8371-826c402525cb\" (UID: \"987893f5-5826-4d00-8371-826c402525cb\") " Oct 29 22:40:35.475848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.475770 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\" (UniqueName: \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\") pod \"084ca685-9944-4791-9dbf-e097c7e8cc10\" (UID: \"084ca685-9944-4791-9dbf-e097c7e8cc10\") " Oct 29 22:40:35.476312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.476164 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.476579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.476546 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.476883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.476863 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.477758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.477729 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.477922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.477909 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.478048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.478036 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.478182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.478169 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.478275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.478265 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.479183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.479156 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.481510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.481432 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.483588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.483545 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:35.488609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.484442 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:35.488609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.484604 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr" pod "084ca685-9944-4791-9dbf-e097c7e8cc10" (UID: "084ca685-9944-4791-9dbf-e097c7e8cc10"). InnerVolumeSpecName "kube-api-access-4v7tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:35.490388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.490321 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk" pod "987893f5-5826-4d00-8371-826c402525cb" (UID: "987893f5-5826-4d00-8371-826c402525cb"). InnerVolumeSpecName "kube-api-access-hvdpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580069 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084ca685-9944-4791-9dbf-e097c7e8cc10-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580117 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7tr\" (UniqueName: \"kubernetes.io/projected/084ca685-9944-4791-9dbf-e097c7e8cc10-kube-api-access-4v7tr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580126 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987893f5-5826-4d00-8371-826c402525cb-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580136 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580145 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987893f5-5826-4d00-8371-826c402525cb-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580151 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580152 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084ca685-9944-4791-9dbf-e097c7e8cc10-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.580465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.580160 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdpk\" (UniqueName: \"kubernetes.io/projected/987893f5-5826-4d00-8371-826c402525cb-kube-api-access-hvdpk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.987609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.987575 4543 generic.go:356] "Generic (PLEG): container finished" podID="987893f5-5826-4d00-8371-826c402525cb" containerID="78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52" exitCode=0 Oct 29 22:40:35.987787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.987625 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" Oct 29 22:40:35.987787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.987655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" event={"ID":"987893f5-5826-4d00-8371-826c402525cb","Type":"ContainerDied","Data":"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52"} Oct 29 22:40:35.987787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.987702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r" event={"ID":"987893f5-5826-4d00-8371-826c402525cb","Type":"ContainerDied","Data":"db5baab582b4fc14d5a5dca2360a2bc0a9532ab85871b5d56422649df77a10ca"} Oct 29 22:40:35.987787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.987722 4543 scope.go:122] "RemoveContainer" containerID="78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52" Oct 29 22:40:35.989536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.989509 4543 generic.go:356] "Generic (PLEG): container finished" podID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerID="533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81" exitCode=0 Oct 29 22:40:35.989683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.989557 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" Oct 29 22:40:35.989683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.989615 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" event={"ID":"084ca685-9944-4791-9dbf-e097c7e8cc10","Type":"ContainerDied","Data":"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81"} Oct 29 22:40:35.989683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:35.989653 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf" event={"ID":"084ca685-9944-4791-9dbf-e097c7e8cc10","Type":"ContainerDied","Data":"9725f8faae6fe28696f77cda7ee1a8239b7a5d965885a7fd40fb1c858dee69df"} Oct 29 22:40:36.001219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.001203 4543 scope.go:122] "RemoveContainer" containerID="78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52" Oct 29 22:40:36.001503 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:36.001483 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52\": container with ID starting with 78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52 not found: ID does not exist" containerID="78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52" Oct 29 22:40:36.001570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.001509 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52"} err="failed to get container status \"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52\": rpc error: code = NotFound desc = could not find container \"78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52\": container with ID starting with 78e59081971104773ec8082af6229c731df4aaf87e988f7e7a069140ac239b52 not found: ID does not exist" Oct 29 22:40:36.001570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.001526 4543 scope.go:122] "RemoveContainer" containerID="533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81" Oct 29 22:40:36.011217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.011198 4543 scope.go:122] "RemoveContainer" containerID="533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81" Oct 29 22:40:36.011472 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:36.011453 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81\": container with ID starting with 533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81 not found: ID does not exist" containerID="533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81" Oct 29 22:40:36.011534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.011480 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81"} err="failed to get container status \"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81\": rpc error: code = NotFound desc = could not find container \"533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81\": container with ID starting with 533b4efaa9579edaa9ee84289b6f60b94f2f6ac82eafc78a4d38b553e95bde81 not found: ID does not exist" Oct 29 22:40:36.049190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.049166 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 22:40:36.062134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.062114 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/premium-w2-92d984-kserve-5c54d8b696-9868r"] Oct 29 22:40:36.091267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.091247 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 22:40:36.096250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.096226 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/simulator-w2-92d984-kserve-c7956f6c7-htxdf"] Oct 29 22:40:36.387667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.387607 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" path="/var/lib/kubelet/pods/084ca685-9944-4791-9dbf-e097c7e8cc10/volumes" Oct 29 22:40:36.388121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:36.388106 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987893f5-5826-4d00-8371-826c402525cb" path="/var/lib/kubelet/pods/987893f5-5826-4d00-8371-826c402525cb/volumes" Oct 29 22:40:40.466661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:40.466628 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84"] Oct 29 22:40:40.470943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:40.470916 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-bef83b-smz84"] Oct 29 22:40:41.969330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:41.969291 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:42.404259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:42.393958 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f0b477-d843-485f-80a0-366a747c38a7" path="/var/lib/kubelet/pods/72f0b477-d843-485f-80a0-366a747c38a7/volumes" Oct 29 22:40:42.974934 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:42.974893 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:44.558779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:44.558747 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl"] Oct 29 22:40:44.586928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:44.586889 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0a3d53-q2cfl"] Oct 29 22:40:46.388978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:46.388942 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64" path="/var/lib/kubelet/pods/2bca4eb5-ccbb-4cea-bbd5-3212ff69dd64/volumes" Oct 29 22:40:47.687187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.687155 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq"] Oct 29 22:40:47.688016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688001 4543 cpu_manager.go:414] "containerMap: removing container" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" Oct 29 22:40:47.688092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688018 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" Oct 29 22:40:47.688092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688040 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e875ebaa-e8ec-46e6-8a38-940676474d71" containerName="revoke-keys" Oct 29 22:40:47.688092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688045 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e875ebaa-e8ec-46e6-8a38-940676474d71" containerName="revoke-keys" Oct 29 22:40:47.688092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688056 4543 cpu_manager.go:414] "containerMap: removing container" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" Oct 29 22:40:47.688092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688061 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688095 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688103 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688112 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688117 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688128 4543 cpu_manager.go:414] "containerMap: removing container" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688132 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688142 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5d528022-6d87-45e8-a500-d60c9e40042a" containerName="revoke-keys" Oct 29 22:40:47.688261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688146 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d528022-6d87-45e8-a500-d60c9e40042a" containerName="revoke-keys" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688274 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="987893f5-5826-4d00-8371-826c402525cb" containerName="main" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688286 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e875ebaa-e8ec-46e6-8a38-940676474d71" containerName="revoke-keys" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688292 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="084ca685-9944-4791-9dbf-e097c7e8cc10" containerName="main" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688299 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688304 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db05227c-5018-4052-a277-4864230a5a65" containerName="istio-proxy" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688310 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f25693-735f-44db-852b-c78561e828de" containerName="payload-processing" Oct 29 22:40:47.688514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.688317 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d528022-6d87-45e8-a500-d60c9e40042a" containerName="revoke-keys" Oct 29 22:40:47.693723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.693706 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.704169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.704147 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv"] Oct 29 22:40:47.710475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.710443 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.712382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.712342 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:40:47.712890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.712864 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:40:47.714672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.714651 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq"] Oct 29 22:40:47.744245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.744218 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv"] Oct 29 22:40:47.816849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.816819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.816979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.816888 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.817062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.817042 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.817195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.817180 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.817234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.817217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpmt\" (UniqueName: \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.817319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.817259 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jt6\" (UniqueName: \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.919041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919015 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.919203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919102 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.919203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919179 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.919324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919233 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.919379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919358 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpmt\" (UniqueName: \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.919440 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jt6\" (UniqueName: \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.919876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919816 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.920288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.919989 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.921802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.921779 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:47.921896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.921876 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.945039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.944971 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpmt\" (UniqueName: \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\") pod \"maas-api-revoke-sub-bcebd115-rwzkq\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:47.945194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:47.945061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jt6\" (UniqueName: \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\") pod \"maas-api-revoke-sub-0a4b68d9-pzxfv\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:48.010432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.010414 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:48.031520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.031496 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:48.333119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.332745 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq"] Oct 29 22:40:48.334198 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:40:48.334153 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff0d4d_ad15_4394_b254_8c1b22cf5b6c.slice/crio-15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de WatchSource:0}: Error finding container 15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de: Status 404 returned error can't find the container with id 15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de Oct 29 22:40:48.408972 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:40:48.408942 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96069476_078a_46bf_90e3_6e996d2fca77.slice/crio-d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828 WatchSource:0}: Error finding container d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828: Status 404 returned error can't find the container with id d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828 Oct 29 22:40:48.412296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.412279 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv"] Oct 29 22:40:48.706341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.706306 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:40:48.707121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:48.706586 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" containerID="cri-o://b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81" gracePeriod=30 Oct 29 22:40:49.168788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.168700 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" event={"ID":"96069476-078a-46bf-90e3-6e996d2fca77","Type":"ContainerStarted","Data":"9776cdfdd3590519583cb4effc62979f90b9f849e8680ab2e67adab1647e9dcc"} Oct 29 22:40:49.168788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.168763 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" event={"ID":"96069476-078a-46bf-90e3-6e996d2fca77","Type":"ContainerStarted","Data":"d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828"} Oct 29 22:40:49.170520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.170488 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" event={"ID":"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c","Type":"ContainerStarted","Data":"9adac73e35b81eea34f1ed8271ae061893efffc2cc07418a7959c0baef82669c"} Oct 29 22:40:49.170647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.170534 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" event={"ID":"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c","Type":"ContainerStarted","Data":"15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de"} Oct 29 22:40:49.216403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.216346 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" podStartSLOduration=2.216335019 podStartE2EDuration="2.216335019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:49.212156041 +0000 UTC m=+3510.954086716" watchObservedRunningTime="2026-10-29 22:40:49.216335019 +0000 UTC m=+3510.958265692" Oct 29 22:40:49.259681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.259639 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" podStartSLOduration=2.259627206 podStartE2EDuration="2.259627206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:49.255286325 +0000 UTC m=+3510.997217002" watchObservedRunningTime="2026-10-29 22:40:49.259627206 +0000 UTC m=+3511.001557879" Oct 29 22:40:49.307548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.307525 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj"] Oct 29 22:40:49.312373 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.312357 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.333550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.333435 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj"] Oct 29 22:40:49.367163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.367133 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jzs\" (UniqueName: \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.367429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.367402 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.367593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.367572 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.469534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.469507 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.469666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.469608 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jzs\" (UniqueName: \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.469844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.469823 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.470342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.470324 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.472426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.472402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.495355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.495336 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jzs\" (UniqueName: \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.628684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.628660 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:49.958335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:49.958152 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj"] Oct 29 22:40:50.179621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:50.179590 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" event={"ID":"06fefc35-4714-46ad-a270-b709973d20b8","Type":"ContainerStarted","Data":"8db12f6b6d603c527a337ce569d27e8bca56d0290dc00277de91bcefaf134584"} Oct 29 22:40:50.181213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:50.181185 4543 generic.go:356] "Generic (PLEG): container finished" podID="96069476-078a-46bf-90e3-6e996d2fca77" containerID="9776cdfdd3590519583cb4effc62979f90b9f849e8680ab2e67adab1647e9dcc" exitCode=0 Oct 29 22:40:50.181334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:50.181260 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" event={"ID":"96069476-078a-46bf-90e3-6e996d2fca77","Type":"ContainerDied","Data":"9776cdfdd3590519583cb4effc62979f90b9f849e8680ab2e67adab1647e9dcc"} Oct 29 22:40:50.183001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:50.182980 4543 generic.go:356] "Generic (PLEG): container finished" podID="67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" containerID="9adac73e35b81eea34f1ed8271ae061893efffc2cc07418a7959c0baef82669c" exitCode=0 Oct 29 22:40:50.183126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:50.183058 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" event={"ID":"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c","Type":"ContainerDied","Data":"9adac73e35b81eea34f1ed8271ae061893efffc2cc07418a7959c0baef82669c"} Oct 29 22:40:51.081401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.081368 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 22:40:51.115343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.115323 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 22:40:51.194852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.194815 4543 generic.go:356] "Generic (PLEG): container finished" podID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerID="7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9" exitCode=0 Oct 29 22:40:51.195020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.194914 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" Oct 29 22:40:51.195020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.194924 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" event={"ID":"162d466f-691b-4409-bd40-fabf0d1f83aa","Type":"ContainerDied","Data":"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9"} Oct 29 22:40:51.195020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.194984 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w" event={"ID":"162d466f-691b-4409-bd40-fabf0d1f83aa","Type":"ContainerDied","Data":"56c9bcb89f96a8e962162f34d34352363f9c4d2705665bc138ca4ef88508e1d6"} Oct 29 22:40:51.195020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.195014 4543 scope.go:122] "RemoveContainer" containerID="7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9" Oct 29 22:40:51.197566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.197540 4543 generic.go:356] "Generic (PLEG): container finished" podID="06fefc35-4714-46ad-a270-b709973d20b8" containerID="a29486058eef3eb6cd8ca2c24467103fb5e80375b605aed5179b5536e543cfc2" exitCode=0 Oct 29 22:40:51.197706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.197627 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" event={"ID":"06fefc35-4714-46ad-a270-b709973d20b8","Type":"ContainerDied","Data":"a29486058eef3eb6cd8ca2c24467103fb5e80375b605aed5179b5536e543cfc2"} Oct 29 22:40:51.199542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.199520 4543 generic.go:356] "Generic (PLEG): container finished" podID="29fc541e-183b-432e-bc68-116f8262171e" containerID="95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9" exitCode=0 Oct 29 22:40:51.199781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.199762 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" Oct 29 22:40:51.199890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.199871 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" event={"ID":"29fc541e-183b-432e-bc68-116f8262171e","Type":"ContainerDied","Data":"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9"} Oct 29 22:40:51.199963 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.199931 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj" event={"ID":"29fc541e-183b-432e-bc68-116f8262171e","Type":"ContainerDied","Data":"2afaf144949869a753b2f20794fed7c7560cea02c9f6878b74ae8e800c2beeb1"} Oct 29 22:40:51.202859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.202832 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\" (UniqueName: \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\") pod \"162d466f-691b-4409-bd40-fabf0d1f83aa\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " Oct 29 22:40:51.202969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.202879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\") pod \"162d466f-691b-4409-bd40-fabf0d1f83aa\" (UID: \"162d466f-691b-4409-bd40-fabf0d1f83aa\") " Oct 29 22:40:51.203052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.202969 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\" (UniqueName: \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\") pod \"29fc541e-183b-432e-bc68-116f8262171e\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " Oct 29 22:40:51.203603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.203559 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume" pod "162d466f-691b-4409-bd40-fabf0d1f83aa" (UID: "162d466f-691b-4409-bd40-fabf0d1f83aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:51.203956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.203919 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume" pod "29fc541e-183b-432e-bc68-116f8262171e" (UID: "29fc541e-183b-432e-bc68-116f8262171e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:51.204124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.203173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\") pod \"29fc541e-183b-432e-bc68-116f8262171e\" (UID: \"29fc541e-183b-432e-bc68-116f8262171e\") " Oct 29 22:40:51.204959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.204937 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162d466f-691b-4409-bd40-fabf0d1f83aa-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.205215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.205200 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fc541e-183b-432e-bc68-116f8262171e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.207596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.207563 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7" pod "162d466f-691b-4409-bd40-fabf0d1f83aa" (UID: "162d466f-691b-4409-bd40-fabf0d1f83aa"). InnerVolumeSpecName "kube-api-access-jbhf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:51.207899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.207872 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v" pod "29fc541e-183b-432e-bc68-116f8262171e" (UID: "29fc541e-183b-432e-bc68-116f8262171e"). InnerVolumeSpecName "kube-api-access-vsn9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:51.217204 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.217186 4543 scope.go:122] "RemoveContainer" containerID="7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9" Oct 29 22:40:51.217542 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:51.217519 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9\": container with ID starting with 7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9 not found: ID does not exist" containerID="7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9" Oct 29 22:40:51.217643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.217552 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9"} err="failed to get container status \"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9\": rpc error: code = NotFound desc = could not find container \"7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9\": container with ID starting with 7d439142820bcb30ef8b20c2688e77ffc6225880888d2fbdf15e6348521e37b9 not found: ID does not exist" Oct 29 22:40:51.217643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.217576 4543 scope.go:122] "RemoveContainer" containerID="95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9" Oct 29 22:40:51.234149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.234130 4543 scope.go:122] "RemoveContainer" containerID="95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9" Oct 29 22:40:51.235432 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:51.235398 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9\": container with ID starting with 95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9 not found: ID does not exist" containerID="95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9" Oct 29 22:40:51.235591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.235435 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9"} err="failed to get container status \"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9\": rpc error: code = NotFound desc = could not find container \"95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9\": container with ID starting with 95c20d94abcf900409a11ccbc02633e0bd08aba56f790bd86d13d938701378d9 not found: ID does not exist" Oct 29 22:40:51.307207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.307139 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhf7\" (UniqueName: \"kubernetes.io/projected/162d466f-691b-4409-bd40-fabf0d1f83aa-kube-api-access-jbhf7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.307207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.307168 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsn9v\" (UniqueName: \"kubernetes.io/projected/29fc541e-183b-432e-bc68-116f8262171e-kube-api-access-vsn9v\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.372614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.372583 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:51.402577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.402554 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:51.510129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.509901 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\") pod \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " Oct 29 22:40:51.510129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510021 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\" (UniqueName: \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\") pod \"96069476-078a-46bf-90e3-6e996d2fca77\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " Oct 29 22:40:51.510129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510061 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\") pod \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " Oct 29 22:40:51.510366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510151 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\") pod \"96069476-078a-46bf-90e3-6e996d2fca77\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " Oct 29 22:40:51.510366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510304 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\" (UniqueName: \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\") pod \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\" (UID: \"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c\") " Oct 29 22:40:51.510366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510338 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\") pod \"96069476-078a-46bf-90e3-6e996d2fca77\" (UID: \"96069476-078a-46bf-90e3-6e996d2fca77\") " Oct 29 22:40:51.510543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510370 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca" pod "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" (UID: "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:51.510688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.510658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca" pod "96069476-078a-46bf-90e3-6e996d2fca77" (UID: "96069476-078a-46bf-90e3-6e996d2fca77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:51.511231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.511213 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.511388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.511364 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.512794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.512770 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt" pod "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" (UID: "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c"). InnerVolumeSpecName "kube-api-access-zvpmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:51.513004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.512982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca" pod "96069476-078a-46bf-90e3-6e996d2fca77" (UID: "96069476-078a-46bf-90e3-6e996d2fca77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:51.513004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.512997 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca" pod "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" (UID: "67ff0d4d-ad15-4394-b254-8c1b22cf5b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:51.513251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.513235 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6" pod "96069476-078a-46bf-90e3-6e996d2fca77" (UID: "96069476-078a-46bf-90e3-6e996d2fca77"). InnerVolumeSpecName "kube-api-access-p8jt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:51.555149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.555126 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 22:40:51.561610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.561561 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-92d984-84c7d9f846-mtl8w"] Oct 29 22:40:51.598972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.598951 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 22:40:51.607160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.607141 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-92d984-6c6b4bf855-j7grj"] Oct 29 22:40:51.612863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.612843 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jt6\" (UniqueName: \"kubernetes.io/projected/96069476-078a-46bf-90e3-6e996d2fca77-kube-api-access-p8jt6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.612965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.612865 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.612965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.612875 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpmt\" (UniqueName: \"kubernetes.io/projected/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c-kube-api-access-zvpmt\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:51.612965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:51.612884 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96069476-078a-46bf-90e3-6e996d2fca77-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:52.227230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.227203 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" Oct 29 22:40:52.227230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.227221 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq" event={"ID":"67ff0d4d-ad15-4394-b254-8c1b22cf5b6c","Type":"ContainerDied","Data":"15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de"} Oct 29 22:40:52.227799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.227267 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15a65292d13892d4df7570132703be8b76fb59868de05dde2c110594c21d32de" Oct 29 22:40:52.229705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.229686 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" Oct 29 22:40:52.229867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.229722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv" event={"ID":"96069476-078a-46bf-90e3-6e996d2fca77","Type":"ContainerDied","Data":"d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828"} Oct 29 22:40:52.229867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.229761 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d933757f251db1c59e551f0ef3ccf6b6c3b84051a9e1065c14b9f09ccd5ae828" Oct 29 22:40:52.387552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.387525 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" path="/var/lib/kubelet/pods/162d466f-691b-4409-bd40-fabf0d1f83aa/volumes" Oct 29 22:40:52.387997 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.387983 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fc541e-183b-432e-bc68-116f8262171e" path="/var/lib/kubelet/pods/29fc541e-183b-432e-bc68-116f8262171e/volumes" Oct 29 22:40:52.397208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.397184 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:52.529296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.529264 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\") pod \"06fefc35-4714-46ad-a270-b709973d20b8\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " Oct 29 22:40:52.529476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.529302 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\" (UniqueName: \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\") pod \"06fefc35-4714-46ad-a270-b709973d20b8\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " Oct 29 22:40:52.529476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.529339 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\") pod \"06fefc35-4714-46ad-a270-b709973d20b8\" (UID: \"06fefc35-4714-46ad-a270-b709973d20b8\") " Oct 29 22:40:52.529705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.529683 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca" pod "06fefc35-4714-46ad-a270-b709973d20b8" (UID: "06fefc35-4714-46ad-a270-b709973d20b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:52.530065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.530046 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:52.531590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.531568 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs" pod "06fefc35-4714-46ad-a270-b709973d20b8" (UID: "06fefc35-4714-46ad-a270-b709973d20b8"). InnerVolumeSpecName "kube-api-access-77jzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:52.531665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.531607 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca" pod "06fefc35-4714-46ad-a270-b709973d20b8" (UID: "06fefc35-4714-46ad-a270-b709973d20b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:52.642811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.642763 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fefc35-4714-46ad-a270-b709973d20b8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:52.643190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:52.643167 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jzs\" (UniqueName: \"kubernetes.io/projected/06fefc35-4714-46ad-a270-b709973d20b8-kube-api-access-77jzs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:53.075416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.075385 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc"] Oct 29 22:40:53.076262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076244 4543 cpu_manager.go:414] "containerMap: removing container" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" Oct 29 22:40:53.076321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076263 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" Oct 29 22:40:53.076321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076304 4543 cpu_manager.go:414] "containerMap: removing container" podUID="06fefc35-4714-46ad-a270-b709973d20b8" containerName="revoke-keys" Oct 29 22:40:53.076321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076310 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fefc35-4714-46ad-a270-b709973d20b8" containerName="revoke-keys" Oct 29 22:40:53.076321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076321 4543 cpu_manager.go:414] "containerMap: removing container" podUID="67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" containerName="revoke-keys" Oct 29 22:40:53.076465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076326 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" containerName="revoke-keys" Oct 29 22:40:53.076465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076339 4543 cpu_manager.go:414] "containerMap: removing container" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" Oct 29 22:40:53.076465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076344 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" Oct 29 22:40:53.076465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076359 4543 cpu_manager.go:414] "containerMap: removing container" podUID="96069476-078a-46bf-90e3-6e996d2fca77" containerName="revoke-keys" Oct 29 22:40:53.076465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076364 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96069476-078a-46bf-90e3-6e996d2fca77" containerName="revoke-keys" Oct 29 22:40:53.076618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076472 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="162d466f-691b-4409-bd40-fabf0d1f83aa" containerName="payload-processing" Oct 29 22:40:53.076618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076487 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fefc35-4714-46ad-a270-b709973d20b8" containerName="revoke-keys" Oct 29 22:40:53.076618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076496 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" containerName="revoke-keys" Oct 29 22:40:53.076618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076502 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fc541e-183b-432e-bc68-116f8262171e" containerName="payload-pre-processing" Oct 29 22:40:53.076618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.076509 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="96069476-078a-46bf-90e3-6e996d2fca77" containerName="revoke-keys" Oct 29 22:40:53.080476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.080454 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.117202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.117162 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc"] Oct 29 22:40:53.239246 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.239216 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" Oct 29 22:40:53.239724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.239258 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj" event={"ID":"06fefc35-4714-46ad-a270-b709973d20b8","Type":"ContainerDied","Data":"8db12f6b6d603c527a337ce569d27e8bca56d0290dc00277de91bcefaf134584"} Oct 29 22:40:53.239724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.239308 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db12f6b6d603c527a337ce569d27e8bca56d0290dc00277de91bcefaf134584" Oct 29 22:40:53.252447 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.252419 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.252556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.252481 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.252681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.252662 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vd4l\" (UniqueName: \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.299463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.299435 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:40:53.299631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.299497 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:40:53.353707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.353652 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vd4l\" (UniqueName: \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.353794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.353752 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.353794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.353785 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.354430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.354409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.356279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.356253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.379183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.379159 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vd4l\" (UniqueName: \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.401236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.401215 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:53.713620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:53.713589 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc"] Oct 29 22:40:53.714301 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:40:53.714277 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780bf253_d43c_4fb5_99c8_cf656e0a5bba.slice/crio-d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a WatchSource:0}: Error finding container d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a: Status 404 returned error can't find the container with id d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a Oct 29 22:40:54.062524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.062499 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:40:54.169493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169462 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169505 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169577 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169610 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169640 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169676 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169713 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169775 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.169938 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169803 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\") pod \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\" (UID: \"af6b5abd-9192-4a00-9ab5-566bcd01dde7\") " Oct 29 22:40:54.170049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.169989 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.170049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170032 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:54.170393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170228 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.170489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170405 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.170583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170560 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.170747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170727 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.170801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170754 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.170801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170770 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.170801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170785 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.171037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.170799 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.172112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.172090 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:54.172190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.172172 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "kube-api-access-ms4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.172517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.172501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:54.172559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.172535 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token" pod "af6b5abd-9192-4a00-9ab5-566bcd01dde7" (UID: "af6b5abd-9192-4a00-9ab5-566bcd01dde7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:54.249951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.249924 4543 generic.go:356] "Generic (PLEG): container finished" podID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerID="b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81" exitCode=0 Oct 29 22:40:54.250344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.249969 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" Oct 29 22:40:54.250344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.250049 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerDied","Data":"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81"} Oct 29 22:40:54.250344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.250137 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g" event={"ID":"af6b5abd-9192-4a00-9ab5-566bcd01dde7","Type":"ContainerDied","Data":"c365366b573257b0400bec8e7320bf8469cddd703724d41963be30787464b3b5"} Oct 29 22:40:54.250344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.250165 4543 scope.go:122] "RemoveContainer" containerID="b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81" Oct 29 22:40:54.251964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.251935 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" event={"ID":"780bf253-d43c-4fb5-99c8-cf656e0a5bba","Type":"ContainerStarted","Data":"d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a"} Oct 29 22:40:54.262277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.262257 4543 scope.go:122] "RemoveContainer" containerID="0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59" Oct 29 22:40:54.272114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272097 4543 scope.go:122] "RemoveContainer" containerID="b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81" Oct 29 22:40:54.272381 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:54.272354 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81\": container with ID starting with b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81 not found: ID does not exist" containerID="b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81" Oct 29 22:40:54.272483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272391 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81"} err="failed to get container status \"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81\": rpc error: code = NotFound desc = could not find container \"b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81\": container with ID starting with b179bfed94a0597696c1571c85a5039db9522e039827acbe90dc049b28ab2f81 not found: ID does not exist" Oct 29 22:40:54.272483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272410 4543 scope.go:122] "RemoveContainer" containerID="0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59" Oct 29 22:40:54.272678 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:54.272663 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59\": container with ID starting with 0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59 not found: ID does not exist" containerID="0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59" Oct 29 22:40:54.272719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272683 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59"} err="failed to get container status \"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59\": rpc error: code = NotFound desc = could not find container \"0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59\": container with ID starting with 0bea9c81be0a02c06c1a82ed7151db08f6d18ad73fcc2217aa282d43989cfa59 not found: ID does not exist" Oct 29 22:40:54.272756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272724 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4nn\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-kube-api-access-ms4nn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.272756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272742 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.272756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272751 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.272857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.272758 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6b5abd-9192-4a00-9ab5-566bcd01dde7-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:54.311520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.311452 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" podStartSLOduration=1.311439927 podStartE2EDuration="1.311439927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:54.310596825 +0000 UTC m=+3516.052527497" watchObservedRunningTime="2026-10-29 22:40:54.311439927 +0000 UTC m=+3516.053370632" Oct 29 22:40:54.355031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.355002 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:40:54.365371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.365345 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-7c25g"] Oct 29 22:40:54.391430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:54.391400 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" path="/var/lib/kubelet/pods/af6b5abd-9192-4a00-9ab5-566bcd01dde7/volumes" Oct 29 22:40:55.263794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:55.263767 4543 generic.go:356] "Generic (PLEG): container finished" podID="780bf253-d43c-4fb5-99c8-cf656e0a5bba" containerID="51e39b1adbf88978c82763ecaa3497c55e1eb4e6d9d67477c527d7e90f862dba" exitCode=0 Oct 29 22:40:55.264202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:55.263841 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" event={"ID":"780bf253-d43c-4fb5-99c8-cf656e0a5bba","Type":"ContainerDied","Data":"51e39b1adbf88978c82763ecaa3497c55e1eb4e6d9d67477c527d7e90f862dba"} Oct 29 22:40:56.476844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.476445 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:56.502701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.502667 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\") pod \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " Oct 29 22:40:56.502896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.502729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\" (UniqueName: \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\") pod \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " Oct 29 22:40:56.502896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.502827 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\") pod \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\" (UID: \"780bf253-d43c-4fb5-99c8-cf656e0a5bba\") " Oct 29 22:40:56.503884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.503419 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca" pod "780bf253-d43c-4fb5-99c8-cf656e0a5bba" (UID: "780bf253-d43c-4fb5-99c8-cf656e0a5bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:56.503884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.503583 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:56.527292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.526636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca" pod "780bf253-d43c-4fb5-99c8-cf656e0a5bba" (UID: "780bf253-d43c-4fb5-99c8-cf656e0a5bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:56.527292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.527110 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l" pod "780bf253-d43c-4fb5-99c8-cf656e0a5bba" (UID: "780bf253-d43c-4fb5-99c8-cf656e0a5bba"). InnerVolumeSpecName "kube-api-access-7vd4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:56.604790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.604736 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vd4l\" (UniqueName: \"kubernetes.io/projected/780bf253-d43c-4fb5-99c8-cf656e0a5bba-kube-api-access-7vd4l\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:56.604790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:56.604772 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780bf253-d43c-4fb5-99c8-cf656e0a5bba-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:40:57.282206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:57.282173 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" Oct 29 22:40:57.282377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:57.282208 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc" event={"ID":"780bf253-d43c-4fb5-99c8-cf656e0a5bba","Type":"ContainerDied","Data":"d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a"} Oct 29 22:40:57.282377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:57.282253 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2fea06657384aee9e94ea53ca0b6e1f4153b844c75e393136595d7fb464750a" Oct 29 22:40:57.971460 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:40:57.971421 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:40:59.995287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:40:59.995223 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:41:00.028205 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:00.028170 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:41:03.672652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:03.672600 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:41:03.698900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:03.698866 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:41:04.996308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:04.996266 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" containerID="cri-o://72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b" gracePeriod=25 Oct 29 22:41:05.029667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.029626 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" containerID="cri-o://61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3" gracePeriod=25 Oct 29 22:41:05.202680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.202650 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:41:05.203463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203448 4543 cpu_manager.go:414] "containerMap: removing container" podUID="780bf253-d43c-4fb5-99c8-cf656e0a5bba" containerName="revoke-keys" Oct 29 22:41:05.203515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203465 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780bf253-d43c-4fb5-99c8-cf656e0a5bba" containerName="revoke-keys" Oct 29 22:41:05.203550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203523 4543 cpu_manager.go:414] "containerMap: removing container" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:05.203550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203527 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:05.203550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203538 4543 cpu_manager.go:414] "containerMap: removing container" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:05.203550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203543 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:05.203687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203644 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:05.203687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.203652 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="780bf253-d43c-4fb5-99c8-cf656e0a5bba" containerName="revoke-keys" Oct 29 22:41:05.207717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.207700 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.265543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.258922 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:41:05.293818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.293782 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcn7w\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.293980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.293938 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294002 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294260 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294299 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294377 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294485 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294548 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.294630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.294576 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.311564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.311539 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-580af2-openshift-default-dockercfg-n7dzr\"" Oct 29 22:41:05.396262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396349 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcn7w\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396492 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396556 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396633 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396687 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396727 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396711 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396737 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396766 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.396904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.396845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.397230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.397036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.397429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.397409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.398885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.398863 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.399120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.399064 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.422386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.422368 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.422544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.422523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcn7w\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.541932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.541871 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:05.878817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.878790 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:41:05.879590 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:41:05.879564 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf253a666_b1cf_48a6_ad54_41276739b5b8.slice/crio-fec8176eb601891fedfdbe76329f491f302550937bef1b6f54c848ddce9097e0 WatchSource:0}: Error finding container fec8176eb601891fedfdbe76329f491f302550937bef1b6f54c848ddce9097e0: Status 404 returned error can't find the container with id fec8176eb601891fedfdbe76329f491f302550937bef1b6f54c848ddce9097e0 Oct 29 22:41:05.882663 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.882627 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:41:05.882753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.882693 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:41:05.882753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:05.882722 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 29 22:41:06.472783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.472740 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" event={"ID":"f253a666-b1cf-48a6-ad54-41276739b5b8","Type":"ContainerStarted","Data":"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff"} Oct 29 22:41:06.473265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.472822 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" event={"ID":"f253a666-b1cf-48a6-ad54-41276739b5b8","Type":"ContainerStarted","Data":"fec8176eb601891fedfdbe76329f491f302550937bef1b6f54c848ddce9097e0"} Oct 29 22:41:06.554342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.554311 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:06.555668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.555635 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:06.555952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.555864 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:06.557000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.556980 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:06.557140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.557033 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:06.557596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:06.557558 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podStartSLOduration=1.557546013 podStartE2EDuration="1.557546013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:06.553476889 +0000 UTC m=+3528.295407574" watchObservedRunningTime="2026-10-29 22:41:06.557546013 +0000 UTC m=+3528.299476686" Oct 29 22:41:07.543203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:07.543165 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:07.543793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:07.543761 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:08.545518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:08.545482 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:08.546171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:08.546140 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:08.675047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:08.675006 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" podUID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" containerName="payload-processing" containerID="cri-o://576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1" gracePeriod=25 Oct 29 22:41:08.699902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:08.699852 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" podUID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" containerName="payload-pre-processing" containerID="cri-o://afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192" gracePeriod=25 Oct 29 22:41:09.083757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.083726 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:41:09.174801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.174759 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\") pod \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " Oct 29 22:41:09.175007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.174846 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\" (UniqueName: \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\") pod \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\" (UID: \"4b7e2b4d-6057-4430-9879-69bc6060fc9e\") " Oct 29 22:41:09.178550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.175553 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume" pod "4b7e2b4d-6057-4430-9879-69bc6060fc9e" (UID: "4b7e2b4d-6057-4430-9879-69bc6060fc9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:09.181231 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.181205 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz" pod "4b7e2b4d-6057-4430-9879-69bc6060fc9e" (UID: "4b7e2b4d-6057-4430-9879-69bc6060fc9e"). InnerVolumeSpecName "kube-api-access-fcjvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:09.213438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.213415 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:41:09.276509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.276474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\") pod \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " Oct 29 22:41:09.276708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.276659 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\" (UniqueName: \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\") pod \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\" (UID: \"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b\") " Oct 29 22:41:09.276969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.276938 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume" pod "77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" (UID: "77e769a2-04c7-4b06-8cff-5ec63fbd2b4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:09.277615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.277577 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e2b4d-6057-4430-9879-69bc6060fc9e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:09.277615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.277609 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjvz\" (UniqueName: \"kubernetes.io/projected/4b7e2b4d-6057-4430-9879-69bc6060fc9e-kube-api-access-fcjvz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:09.277798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.277624 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:09.279269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.279239 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27" pod "77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" (UID: "77e769a2-04c7-4b06-8cff-5ec63fbd2b4b"). InnerVolumeSpecName "kube-api-access-b8v27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:09.378519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.378452 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8v27\" (UniqueName: \"kubernetes.io/projected/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b-kube-api-access-b8v27\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:09.515310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.515276 4543 generic.go:356] "Generic (PLEG): container finished" podID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" containerID="576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1" exitCode=0 Oct 29 22:41:09.515466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.515316 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" Oct 29 22:41:09.515466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.515366 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" event={"ID":"4b7e2b4d-6057-4430-9879-69bc6060fc9e","Type":"ContainerDied","Data":"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1"} Oct 29 22:41:09.515466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.515419 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2" event={"ID":"4b7e2b4d-6057-4430-9879-69bc6060fc9e","Type":"ContainerDied","Data":"aae872f377386830093c1cbd619594e231ba2e55492a46b3c7a23259052ab963"} Oct 29 22:41:09.515466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.515451 4543 scope.go:122] "RemoveContainer" containerID="576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1" Oct 29 22:41:09.517221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.517196 4543 generic.go:356] "Generic (PLEG): container finished" podID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" containerID="afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192" exitCode=0 Oct 29 22:41:09.517363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.517226 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" event={"ID":"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b","Type":"ContainerDied","Data":"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192"} Oct 29 22:41:09.517363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.517241 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" Oct 29 22:41:09.517363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.517281 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89" event={"ID":"77e769a2-04c7-4b06-8cff-5ec63fbd2b4b","Type":"ContainerDied","Data":"51e5acd41cd9de99d90813a97d7e8d0b6e792c1f374f56809830a5d19e9c3987"} Oct 29 22:41:09.547097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.545197 4543 scope.go:122] "RemoveContainer" containerID="576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1" Oct 29 22:41:09.554130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.549382 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:09.554130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.549446 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:09.554433 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:09.554404 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1\": container with ID starting with 576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1 not found: ID does not exist" containerID="576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1" Oct 29 22:41:09.554499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.554448 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1"} err="failed to get container status \"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1\": rpc error: code = NotFound desc = could not find container \"576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1\": container with ID starting with 576f2593ce0f19735e259c469e95656f4a00c5dae4d859fa5fcc621586b734d1 not found: ID does not exist" Oct 29 22:41:09.554499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.554472 4543 scope.go:122] "RemoveContainer" containerID="afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192" Oct 29 22:41:09.566615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.566593 4543 scope.go:122] "RemoveContainer" containerID="afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192" Oct 29 22:41:09.566881 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:09.566860 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192\": container with ID starting with afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192 not found: ID does not exist" containerID="afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192" Oct 29 22:41:09.566954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.566889 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192"} err="failed to get container status \"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192\": rpc error: code = NotFound desc = could not find container \"afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192\": container with ID starting with afc59d86892c6de642abb19a34b776294bd68fc3d5937915cf2847250ffa1192 not found: ID does not exist" Oct 29 22:41:09.587619 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.587600 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:41:09.594183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.594164 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-798a8f-857759587f-7vk89"] Oct 29 22:41:09.655291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.655212 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:41:09.659171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:09.659147 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-798a8f-7997444d58-gwdc2"] Oct 29 22:41:10.388012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:10.387981 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" path="/var/lib/kubelet/pods/4b7e2b4d-6057-4430-9879-69bc6060fc9e/volumes" Oct 29 22:41:10.388487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:10.388472 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" path="/var/lib/kubelet/pods/77e769a2-04c7-4b06-8cff-5ec63fbd2b4b/volumes" Oct 29 22:41:10.553114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:10.551481 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:10.553114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:10.551533 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:11.553247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:11.553195 4543 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.553796 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:11.553257 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.183:15021/healthz/ready\": dial tcp 10.133.1.183:15021: connect: connection refused" Oct 29 22:41:12.547517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:12.547488 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:12.547966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:12.547943 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:12.548625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:12.548604 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:41:14.892581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:14.892271 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz"] Oct 29 22:41:14.919147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:14.918982 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4c1015-mklkz"] Oct 29 22:41:16.395281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:16.395244 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e875ebaa-e8ec-46e6-8a38-940676474d71" path="/var/lib/kubelet/pods/e875ebaa-e8ec-46e6-8a38-940676474d71/volumes" Oct 29 22:41:16.554667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:16.554630 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:16.555387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:16.555360 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:23.225513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.225480 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct"] Oct 29 22:41:23.226295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226278 4543 cpu_manager.go:414] "containerMap: removing container" podUID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" containerName="payload-pre-processing" Oct 29 22:41:23.226355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226296 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" containerName="payload-pre-processing" Oct 29 22:41:23.226355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226308 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" containerName="payload-processing" Oct 29 22:41:23.226355 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226313 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" containerName="payload-processing" Oct 29 22:41:23.226493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226482 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e769a2-04c7-4b06-8cff-5ec63fbd2b4b" containerName="payload-pre-processing" Oct 29 22:41:23.226539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226497 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7e2b4d-6057-4430-9879-69bc6060fc9e" containerName="payload-processing" Oct 29 22:41:23.226539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.226510 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6b5abd-9192-4a00-9ab5-566bcd01dde7" containerName="istio-proxy" Oct 29 22:41:23.230429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.230412 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.248660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.248634 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct"] Oct 29 22:41:23.287554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.287528 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:41:23.287725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.287601 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:41:23.287915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.287895 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.287995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.287966 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bttg\" (UniqueName: \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.288121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.288108 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.389538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.389513 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.389641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.389588 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bttg\" (UniqueName: \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.389684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.389639 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.390195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.390176 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.392116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.392098 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.415776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.415750 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bttg\" (UniqueName: \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.546897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.546843 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:23.861264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:23.861192 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct"] Oct 29 22:41:23.865546 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:41:23.865515 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec32fb9_f848_4734_a77f_12487b915ac9.slice/crio-c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5 WatchSource:0}: Error finding container c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5: Status 404 returned error can't find the container with id c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5 Oct 29 22:41:24.667161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:24.667128 4543 scope.go:122] "RemoveContainer" containerID="b626033d76fcb0ee4a8f275040be3c4d79702485346e9e13611467cacf256a41" Oct 29 22:41:24.680184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:24.680165 4543 scope.go:122] "RemoveContainer" containerID="2415f582f2a17277187e109a39c1f320801f95d0fdc13a0c4d71f84a523e7d55" Oct 29 22:41:24.706486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:24.706455 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" event={"ID":"9ec32fb9-f848-4734-a77f-12487b915ac9","Type":"ContainerStarted","Data":"41dfda38fe2784123d66cf9806ec15b2c0b948ebe2c78bf9cd5125ecf62753b2"} Oct 29 22:41:24.706589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:24.706516 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" event={"ID":"9ec32fb9-f848-4734-a77f-12487b915ac9","Type":"ContainerStarted","Data":"c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5"} Oct 29 22:41:24.751793 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:24.751743 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" podStartSLOduration=1.751725896 podStartE2EDuration="1.751725896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:24.749772181 +0000 UTC m=+3546.491702854" watchObservedRunningTime="2026-10-29 22:41:24.751725896 +0000 UTC m=+3546.493656573" Oct 29 22:41:25.320235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.320205 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:41:25.342194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.342170 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:41:25.431931 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.431898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\") pod \"16b748bc-aa36-4543-b56a-519ccdc8e61f\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " Oct 29 22:41:25.432051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.431948 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\" (UniqueName: \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\") pod \"16b748bc-aa36-4543-b56a-519ccdc8e61f\" (UID: \"16b748bc-aa36-4543-b56a-519ccdc8e61f\") " Oct 29 22:41:25.432051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432036 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\" (UniqueName: \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\") pod \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " Oct 29 22:41:25.432197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432183 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\") pod \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\" (UID: \"4b387aab-8504-45c5-8fb8-da1667bc3d7b\") " Oct 29 22:41:25.432337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432310 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume" pod "16b748bc-aa36-4543-b56a-519ccdc8e61f" (UID: "16b748bc-aa36-4543-b56a-519ccdc8e61f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:25.432574 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432542 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume" pod "4b387aab-8504-45c5-8fb8-da1667bc3d7b" (UID: "4b387aab-8504-45c5-8fb8-da1667bc3d7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:25.432944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432923 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b387aab-8504-45c5-8fb8-da1667bc3d7b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:25.433019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.432950 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b748bc-aa36-4543-b56a-519ccdc8e61f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:25.435028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.435001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk" pod "16b748bc-aa36-4543-b56a-519ccdc8e61f" (UID: "16b748bc-aa36-4543-b56a-519ccdc8e61f"). InnerVolumeSpecName "kube-api-access-6bvkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:25.435143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.435040 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj" pod "4b387aab-8504-45c5-8fb8-da1667bc3d7b" (UID: "4b387aab-8504-45c5-8fb8-da1667bc3d7b"). InnerVolumeSpecName "kube-api-access-fbbhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:25.534400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.534378 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvkk\" (UniqueName: \"kubernetes.io/projected/16b748bc-aa36-4543-b56a-519ccdc8e61f-kube-api-access-6bvkk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:25.534400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.534402 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbhj\" (UniqueName: \"kubernetes.io/projected/4b387aab-8504-45c5-8fb8-da1667bc3d7b-kube-api-access-fbbhj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:25.717860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.717832 4543 generic.go:356] "Generic (PLEG): container finished" podID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerID="72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b" exitCode=0 Oct 29 22:41:25.718273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.717929 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" Oct 29 22:41:25.718273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.717945 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" event={"ID":"16b748bc-aa36-4543-b56a-519ccdc8e61f","Type":"ContainerDied","Data":"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b"} Oct 29 22:41:25.718273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.717995 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9" event={"ID":"16b748bc-aa36-4543-b56a-519ccdc8e61f","Type":"ContainerDied","Data":"4eef7937ed8f0ea727a88c31d7d3d769f3733145984620797bdbcec17e3778be"} Oct 29 22:41:25.718273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.718020 4543 scope.go:122] "RemoveContainer" containerID="72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b" Oct 29 22:41:25.720364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.720339 4543 generic.go:356] "Generic (PLEG): container finished" podID="9ec32fb9-f848-4734-a77f-12487b915ac9" containerID="41dfda38fe2784123d66cf9806ec15b2c0b948ebe2c78bf9cd5125ecf62753b2" exitCode=0 Oct 29 22:41:25.720460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.720415 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" event={"ID":"9ec32fb9-f848-4734-a77f-12487b915ac9","Type":"ContainerDied","Data":"41dfda38fe2784123d66cf9806ec15b2c0b948ebe2c78bf9cd5125ecf62753b2"} Oct 29 22:41:25.722210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.722189 4543 generic.go:356] "Generic (PLEG): container finished" podID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerID="61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3" exitCode=0 Oct 29 22:41:25.722310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.722239 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" Oct 29 22:41:25.722310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.722277 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" event={"ID":"4b387aab-8504-45c5-8fb8-da1667bc3d7b","Type":"ContainerDied","Data":"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3"} Oct 29 22:41:25.722406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.722327 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b" event={"ID":"4b387aab-8504-45c5-8fb8-da1667bc3d7b","Type":"ContainerDied","Data":"b24e4fe37bf430bdefdc9c3e6eed17fb700ca05c1e6b79f4eb30f49e8c702b90"} Oct 29 22:41:25.731125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.730985 4543 scope.go:122] "RemoveContainer" containerID="72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b" Oct 29 22:41:25.731404 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:25.731385 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b\": container with ID starting with 72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b not found: ID does not exist" containerID="72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b" Oct 29 22:41:25.731465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.731409 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b"} err="failed to get container status \"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b\": rpc error: code = NotFound desc = could not find container \"72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b\": container with ID starting with 72276765b11c7609c0a0cdf3e2c3f43f57f300b9b1ccab78d0afd8277352c37b not found: ID does not exist" Oct 29 22:41:25.731465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.731426 4543 scope.go:122] "RemoveContainer" containerID="61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3" Oct 29 22:41:25.741820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.741802 4543 scope.go:122] "RemoveContainer" containerID="61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3" Oct 29 22:41:25.742112 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:25.742063 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3\": container with ID starting with 61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3 not found: ID does not exist" containerID="61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3" Oct 29 22:41:25.742159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.742121 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3"} err="failed to get container status \"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3\": rpc error: code = NotFound desc = could not find container \"61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3\": container with ID starting with 61e268d14faf2ac7bf54715dba476a4a786aaf1c17530bdf397beee7562c56e3 not found: ID does not exist" Oct 29 22:41:25.777721 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.777704 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:41:25.782957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.782936 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-675bf4-68c6d44fd-gsjf9"] Oct 29 22:41:25.821232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.821209 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:41:25.828526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:25.828504 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-675bf4-7d89c87799-6kb2b"] Oct 29 22:41:26.390744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.390712 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" path="/var/lib/kubelet/pods/16b748bc-aa36-4543-b56a-519ccdc8e61f/volumes" Oct 29 22:41:26.391181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.391166 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" path="/var/lib/kubelet/pods/4b387aab-8504-45c5-8fb8-da1667bc3d7b/volumes" Oct 29 22:41:26.897106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.897067 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:26.952202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.952173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\") pod \"9ec32fb9-f848-4734-a77f-12487b915ac9\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " Oct 29 22:41:26.952378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.952294 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\") pod \"9ec32fb9-f848-4734-a77f-12487b915ac9\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " Oct 29 22:41:26.952470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.952452 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\" (UniqueName: \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\") pod \"9ec32fb9-f848-4734-a77f-12487b915ac9\" (UID: \"9ec32fb9-f848-4734-a77f-12487b915ac9\") " Oct 29 22:41:26.952846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.952573 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca" pod "9ec32fb9-f848-4734-a77f-12487b915ac9" (UID: "9ec32fb9-f848-4734-a77f-12487b915ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:26.953304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.953277 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:26.955223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.955195 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca" pod "9ec32fb9-f848-4734-a77f-12487b915ac9" (UID: "9ec32fb9-f848-4734-a77f-12487b915ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:26.956509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:26.956467 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg" pod "9ec32fb9-f848-4734-a77f-12487b915ac9" (UID: "9ec32fb9-f848-4734-a77f-12487b915ac9"). InnerVolumeSpecName "kube-api-access-4bttg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:27.054928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:27.054869 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bttg\" (UniqueName: \"kubernetes.io/projected/9ec32fb9-f848-4734-a77f-12487b915ac9-kube-api-access-4bttg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:27.054928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:27.054897 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec32fb9-f848-4734-a77f-12487b915ac9-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:27.742884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:27.742852 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" Oct 29 22:41:27.743039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:27.742880 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct" event={"ID":"9ec32fb9-f848-4734-a77f-12487b915ac9","Type":"ContainerDied","Data":"c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5"} Oct 29 22:41:27.743039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:27.742937 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c239a352ddf04cb24a7328dfedf4b499ce7477a13ccba5f8d9748ef48ca335b5" Oct 29 22:41:27.962734 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:27.962702 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:41:33.543474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:33.543383 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:41:33.582636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:33.582604 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:41:38.512382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:38.512349 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj"] Oct 29 22:41:38.517254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:38.517232 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-92d984-h7zjj"] Oct 29 22:41:38.544582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:38.544553 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" containerID="cri-o://c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a" gracePeriod=25 Oct 29 22:41:38.583620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:38.583589 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" containerID="cri-o://d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7" gracePeriod=25 Oct 29 22:41:40.060947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:40.060902 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:40.061394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:40.060952 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:40.390605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:40.390523 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d528022-6d87-45e8-a500-d60c9e40042a" path="/var/lib/kubelet/pods/5d528022-6d87-45e8-a500-d60c9e40042a/volumes" Oct 29 22:41:41.168184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:41.168143 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc"] Oct 29 22:41:41.180726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:41.180702 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-798a8f-tt8jc"] Oct 29 22:41:42.386941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:42.386902 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780bf253-d43c-4fb5-99c8-cf656e0a5bba" path="/var/lib/kubelet/pods/780bf253-d43c-4fb5-99c8-cf656e0a5bba/volumes" Oct 29 22:41:43.399327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:43.399291 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj"] Oct 29 22:41:43.414108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:43.414052 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-675bf4-hkmdj"] Oct 29 22:41:44.388087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:44.388049 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fefc35-4714-46ad-a270-b709973d20b8" path="/var/lib/kubelet/pods/06fefc35-4714-46ad-a270-b709973d20b8/volumes" Oct 29 22:41:46.444944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:46.444855 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 22:41:46.445618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:46.445585 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" containerID="cri-o://55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d" gracePeriod=30 Oct 29 22:41:46.823999 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:46.823960 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:41:46.824182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:46.824017 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:41:49.147890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.147851 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:41:49.148386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.148125 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" containerID="cri-o://cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1" gracePeriod=30 Oct 29 22:41:49.892252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.892218 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v"] Oct 29 22:41:49.893020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893003 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" Oct 29 22:41:49.893112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893022 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" Oct 29 22:41:49.893112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893042 4543 cpu_manager.go:414] "containerMap: removing container" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" Oct 29 22:41:49.893112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893047 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" Oct 29 22:41:49.893112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893062 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9ec32fb9-f848-4734-a77f-12487b915ac9" containerName="revoke-keys" Oct 29 22:41:49.893112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893067 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec32fb9-f848-4734-a77f-12487b915ac9" containerName="revoke-keys" Oct 29 22:41:49.893299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893208 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b387aab-8504-45c5-8fb8-da1667bc3d7b" containerName="payload-pre-processing" Oct 29 22:41:49.893299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893219 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b748bc-aa36-4543-b56a-519ccdc8e61f" containerName="payload-processing" Oct 29 22:41:49.893299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.893227 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec32fb9-f848-4734-a77f-12487b915ac9" containerName="revoke-keys" Oct 29 22:41:49.898566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.898549 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:49.927950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.927924 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v"] Oct 29 22:41:49.939519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.937493 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:41:49.939519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.937815 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:41:49.980153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.980127 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5j9\" (UniqueName: \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:49.980303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.980272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:49.980515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:49.980489 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.062582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.062552 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:50.062582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.062568 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:41:50.081603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.081575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.081714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.081704 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.081770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.081759 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5j9\" (UniqueName: \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.082380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.082322 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.084114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.084093 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.107378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.107357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5j9\" (UniqueName: \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\") pod \"maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.212982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.212961 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:50.549321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:50.549292 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v"] Oct 29 22:41:50.550575 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:41:50.550547 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75043950_90de_4326_a750_5fc8a71383b0.slice/crio-be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66 WatchSource:0}: Error finding container be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66: Status 404 returned error can't find the container with id be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66 Oct 29 22:41:51.081139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.081103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" event={"ID":"75043950-90de-4326-a750-5fc8a71383b0","Type":"ContainerStarted","Data":"321b4ddf1b296010b93a8652040b103acdca8bcf31a002cd7a37602c2bbf0f84"} Oct 29 22:41:51.081337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.081165 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" event={"ID":"75043950-90de-4326-a750-5fc8a71383b0","Type":"ContainerStarted","Data":"be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66"} Oct 29 22:41:51.127610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.127525 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" podStartSLOduration=2.127512529 podStartE2EDuration="2.127512529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:51.125149292 +0000 UTC m=+3572.867079964" watchObservedRunningTime="2026-10-29 22:41:51.127512529 +0000 UTC m=+3572.869443202" Oct 29 22:41:51.500453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.500414 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:41:51.500959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.500673 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" containerID="cri-o://237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac" gracePeriod=30 Oct 29 22:41:51.905982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:51.905955 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 22:41:52.014914 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.014879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.014925 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015029 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015088 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015126 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015169 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015175 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.015465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015321 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.015465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015425 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.015705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015466 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\") pod \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\" (UID: \"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701\") " Oct 29 22:41:52.015827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015704 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.015827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.015731 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:52.016457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.016434 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.016580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.016465 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.016580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.016480 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.016580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.016495 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.016580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.016511 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.017485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.017460 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.017726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.017706 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "kube-api-access-h4r7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:52.017801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.017741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:52.018115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.018066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo" pod "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" (UID: "e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:52.093019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.092948 4543 generic.go:356] "Generic (PLEG): container finished" podID="75043950-90de-4326-a750-5fc8a71383b0" containerID="321b4ddf1b296010b93a8652040b103acdca8bcf31a002cd7a37602c2bbf0f84" exitCode=0 Oct 29 22:41:52.093175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.093039 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" event={"ID":"75043950-90de-4326-a750-5fc8a71383b0","Type":"ContainerDied","Data":"321b4ddf1b296010b93a8652040b103acdca8bcf31a002cd7a37602c2bbf0f84"} Oct 29 22:41:52.094668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.094646 4543 generic.go:356] "Generic (PLEG): container finished" podID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerID="55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d" exitCode=0 Oct 29 22:41:52.094735 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.094717 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" Oct 29 22:41:52.094791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.094729 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" event={"ID":"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701","Type":"ContainerDied","Data":"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d"} Oct 29 22:41:52.094791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.094775 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv" event={"ID":"e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701","Type":"ContainerDied","Data":"bbf6a0900f0eb2666d0dfe47f1742115328ad7f108125691ef3b78216024b9d8"} Oct 29 22:41:52.094883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.094797 4543 scope.go:122] "RemoveContainer" containerID="55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d" Oct 29 22:41:52.117119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117064 4543 scope.go:122] "RemoveContainer" containerID="55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d" Oct 29 22:41:52.117370 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:52.117350 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d\": container with ID starting with 55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d not found: ID does not exist" containerID="55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d" Oct 29 22:41:52.117446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117376 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d"} err="failed to get container status \"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d\": rpc error: code = NotFound desc = could not find container \"55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d\": container with ID starting with 55c8f5682d057b80a9de8157f1e1e94f5ce5d302037d4af67b8fb0d49cac570d not found: ID does not exist" Oct 29 22:41:52.117863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117845 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.117901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117868 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.117901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117880 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.117901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.117888 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4r7f\" (UniqueName: \"kubernetes.io/projected/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701-kube-api-access-h4r7f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.183885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.183858 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 22:41:52.190556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.190531 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-bf8lv"] Oct 29 22:41:52.293892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.293864 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7"] Oct 29 22:41:52.294769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.294746 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" Oct 29 22:41:52.294769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.294769 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" Oct 29 22:41:52.294957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.294943 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" containerName="istio-proxy" Oct 29 22:41:52.300330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.300306 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.321191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.321162 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7"] Oct 29 22:41:52.409392 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.409318 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701" path="/var/lib/kubelet/pods/e3d6ee3d-b0d6-4b9f-8c54-f2b2cb307701/volumes" Oct 29 22:41:52.421958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.421934 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.422175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.422155 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.422244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.422187 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ng2\" (UniqueName: \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.523942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.523910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.524386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.523951 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ng2\" (UniqueName: \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.524386 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.524118 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.524730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.524709 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.526791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.526762 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.550164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.550139 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ng2\" (UniqueName: \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\") pod \"maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.615749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.615727 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:52.929454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:52.929430 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7"] Oct 29 22:41:52.930133 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:41:52.930109 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805efdc8_a578_4094_928c_4a660bab9cb7.slice/crio-301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322 WatchSource:0}: Error finding container 301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322: Status 404 returned error can't find the container with id 301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322 Oct 29 22:41:53.105517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.105485 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" event={"ID":"805efdc8-a578-4094-928c-4a660bab9cb7","Type":"ContainerStarted","Data":"301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322"} Oct 29 22:41:53.272930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.272896 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:53.334530 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.334506 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\" (UniqueName: \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\") pod \"75043950-90de-4326-a750-5fc8a71383b0\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " Oct 29 22:41:53.334678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.334594 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\") pod \"75043950-90de-4326-a750-5fc8a71383b0\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " Oct 29 22:41:53.334742 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.334723 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\") pod \"75043950-90de-4326-a750-5fc8a71383b0\" (UID: \"75043950-90de-4326-a750-5fc8a71383b0\") " Oct 29 22:41:53.335147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.335121 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca" pod "75043950-90de-4326-a750-5fc8a71383b0" (UID: "75043950-90de-4326-a750-5fc8a71383b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:53.335450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.335427 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75043950-90de-4326-a750-5fc8a71383b0-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:53.337419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.337388 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9" pod "75043950-90de-4326-a750-5fc8a71383b0" (UID: "75043950-90de-4326-a750-5fc8a71383b0"). InnerVolumeSpecName "kube-api-access-9z5j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:53.338065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.338030 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca" pod "75043950-90de-4326-a750-5fc8a71383b0" (UID: "75043950-90de-4326-a750-5fc8a71383b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:53.436850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.436828 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5j9\" (UniqueName: \"kubernetes.io/projected/75043950-90de-4326-a750-5fc8a71383b0-kube-api-access-9z5j9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:53.436850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:53.436851 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75043950-90de-4326-a750-5fc8a71383b0-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.117211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.117127 4543 generic.go:356] "Generic (PLEG): container finished" podID="805efdc8-a578-4094-928c-4a660bab9cb7" containerID="cdf14737a95d95ea352972210951f95e1342893f3691088aa812c7830d063fd6" exitCode=0 Oct 29 22:41:54.117607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.117219 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" event={"ID":"805efdc8-a578-4094-928c-4a660bab9cb7","Type":"ContainerDied","Data":"cdf14737a95d95ea352972210951f95e1342893f3691088aa812c7830d063fd6"} Oct 29 22:41:54.118952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.118931 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" Oct 29 22:41:54.119116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.118978 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v" event={"ID":"75043950-90de-4326-a750-5fc8a71383b0","Type":"ContainerDied","Data":"be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66"} Oct 29 22:41:54.119116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.119019 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be36ea048e82a944d3a3fa8083e5a3a1cea8353fc2affe9f21692ae56db18f66" Oct 29 22:41:54.436438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.436414 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:41:54.552964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.552933 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.552964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.552969 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553051 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553189 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553243 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553290 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553323 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:54.553437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\") pod \"c513c5dc-9256-40cf-a616-ada014206f81\" (UID: \"c513c5dc-9256-40cf-a616-ada014206f81\") " Oct 29 22:41:54.553610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553557 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:54.553659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553611 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:54.553659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.553637 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:54.554032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:54.554190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554148 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.554190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554161 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513c5dc-9256-40cf-a616-ada014206f81-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.554190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554170 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.554190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554178 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.554190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.554187 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.555778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.555751 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "kube-api-access-2hzfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:54.555892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.555789 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:54.555962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.555932 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:54.555962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.555951 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo" pod "c513c5dc-9256-40cf-a616-ada014206f81" (UID: "c513c5dc-9256-40cf-a616-ada014206f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:54.655609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.655557 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzfr\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-kube-api-access-2hzfr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.655609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.655578 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513c5dc-9256-40cf-a616-ada014206f81-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.655609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.655588 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513c5dc-9256-40cf-a616-ada014206f81-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.655609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.655596 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513c5dc-9256-40cf-a616-ada014206f81-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:54.666818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:54.666796 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 22:41:55.129301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.129266 4543 generic.go:356] "Generic (PLEG): container finished" podID="c513c5dc-9256-40cf-a616-ada014206f81" containerID="cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1" exitCode=0 Oct 29 22:41:55.129767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.129301 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerDied","Data":"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1"} Oct 29 22:41:55.129767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.129328 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" Oct 29 22:41:55.129767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.129371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp" event={"ID":"c513c5dc-9256-40cf-a616-ada014206f81","Type":"ContainerDied","Data":"19f50e2fbfd4342cb25ba1161ecb3c2c006699a1e1b0779f6029178d3519e6c5"} Oct 29 22:41:55.129767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.129402 4543 scope.go:122] "RemoveContainer" containerID="cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1" Oct 29 22:41:55.141745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.141726 4543 scope.go:122] "RemoveContainer" containerID="41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc" Oct 29 22:41:55.154164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.154139 4543 scope.go:122] "RemoveContainer" containerID="cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1" Oct 29 22:41:55.154459 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:55.154439 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1\": container with ID starting with cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1 not found: ID does not exist" containerID="cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1" Oct 29 22:41:55.154512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.154470 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1"} err="failed to get container status \"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1\": rpc error: code = NotFound desc = could not find container \"cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1\": container with ID starting with cf3498330076ac567c40e53043b656bdb9a1c00a3bff7a9ee2575f19b20af6a1 not found: ID does not exist" Oct 29 22:41:55.154512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.154492 4543 scope.go:122] "RemoveContainer" containerID="41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc" Oct 29 22:41:55.154878 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:55.154859 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc\": container with ID starting with 41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc not found: ID does not exist" containerID="41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc" Oct 29 22:41:55.154929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.154886 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc"} err="failed to get container status \"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc\": rpc error: code = NotFound desc = could not find container \"41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc\": container with ID starting with 41a02778ba655e104cb4a29e29089709d41f9cbff8f066ecca2c0596ec49bffc not found: ID does not exist" Oct 29 22:41:55.219217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.219170 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:41:55.232810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.232776 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-798a8f-openshift-default-7dd6f6cf7c-8rsvp"] Oct 29 22:41:55.304499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.304478 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:55.365908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.365879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\") pod \"805efdc8-a578-4094-928c-4a660bab9cb7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " Oct 29 22:41:55.366140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.365937 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\" (UniqueName: \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\") pod \"805efdc8-a578-4094-928c-4a660bab9cb7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " Oct 29 22:41:55.366140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.366029 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\") pod \"805efdc8-a578-4094-928c-4a660bab9cb7\" (UID: \"805efdc8-a578-4094-928c-4a660bab9cb7\") " Oct 29 22:41:55.366305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.366278 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca" pod "805efdc8-a578-4094-928c-4a660bab9cb7" (UID: "805efdc8-a578-4094-928c-4a660bab9cb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:55.366883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.366857 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:55.369053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.369023 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca" pod "805efdc8-a578-4094-928c-4a660bab9cb7" (UID: "805efdc8-a578-4094-928c-4a660bab9cb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:55.369323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.369293 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2" pod "805efdc8-a578-4094-928c-4a660bab9cb7" (UID: "805efdc8-a578-4094-928c-4a660bab9cb7"). InnerVolumeSpecName "kube-api-access-97ng2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:55.468169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.468148 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805efdc8-a578-4094-928c-4a660bab9cb7-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:55.468169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:55.468170 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ng2\" (UniqueName: \"kubernetes.io/projected/805efdc8-a578-4094-928c-4a660bab9cb7-kube-api-access-97ng2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.140272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.140242 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" Oct 29 22:41:56.140272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.140258 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7" event={"ID":"805efdc8-a578-4094-928c-4a660bab9cb7","Type":"ContainerDied","Data":"301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322"} Oct 29 22:41:56.140705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.140296 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="301a361a86ab96004ddf702e7236dac97d51cb5903b6647dc2c17ef81d8f5322" Oct 29 22:41:56.387731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.387695 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c513c5dc-9256-40cf-a616-ada014206f81" path="/var/lib/kubelet/pods/c513c5dc-9256-40cf-a616-ada014206f81/volumes" Oct 29 22:41:56.790780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.790755 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:41:56.887045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887003 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887051 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887128 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887157 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887197 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887254 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887364 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887462 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\") pod \"74f04497-2263-427b-8521-d79e20303f98\" (UID: \"74f04497-2263-427b-8521-d79e20303f98\") " Oct 29 22:41:56.887810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.887779 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:56.888039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888009 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:56.888268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888243 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:56.888358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888274 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:56.888465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888442 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.888538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888473 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.888538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888488 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.888538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888503 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.888744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.888722 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:56.892117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.890733 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:56.892117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.891191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:56.892117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.891203 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:56.892117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.891305 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9" pod "74f04497-2263-427b-8521-d79e20303f98" (UID: "74f04497-2263-427b-8521-d79e20303f98"). InnerVolumeSpecName "kube-api-access-rkcq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:56.990281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.990260 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f04497-2263-427b-8521-d79e20303f98-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.990281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.990283 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f04497-2263-427b-8521-d79e20303f98-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.990418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.990293 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcq9\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-kube-api-access-rkcq9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.990418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.990304 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f04497-2263-427b-8521-d79e20303f98-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:56.990418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:56.990312 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f04497-2263-427b-8521-d79e20303f98-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:57.008053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.008029 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 22:41:57.036760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.036704 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 22:41:57.161815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.161775 4543 generic.go:356] "Generic (PLEG): container finished" podID="74f04497-2263-427b-8521-d79e20303f98" containerID="237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac" exitCode=0 Oct 29 22:41:57.162303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.161831 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" Oct 29 22:41:57.162303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.161859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerDied","Data":"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac"} Oct 29 22:41:57.162303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.161923 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf" event={"ID":"74f04497-2263-427b-8521-d79e20303f98","Type":"ContainerDied","Data":"fc3f72a34f17e6aaef2a9b297be84c5d366e9ff77cd5bd3d41f5aa2cae76eee4"} Oct 29 22:41:57.162303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.161950 4543 scope.go:122] "RemoveContainer" containerID="237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac" Oct 29 22:41:57.175657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.175625 4543 scope.go:122] "RemoveContainer" containerID="f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e" Oct 29 22:41:57.190279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.190257 4543 scope.go:122] "RemoveContainer" containerID="237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac" Oct 29 22:41:57.190620 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:57.190596 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac\": container with ID starting with 237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac not found: ID does not exist" containerID="237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac" Oct 29 22:41:57.190749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.190628 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac"} err="failed to get container status \"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac\": rpc error: code = NotFound desc = could not find container \"237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac\": container with ID starting with 237deb9b5d9eb94ec6947918d88bd307277e90bf3f6063596232d8ff7d6a57ac not found: ID does not exist" Oct 29 22:41:57.190749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.190652 4543 scope.go:122] "RemoveContainer" containerID="f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e" Oct 29 22:41:57.191010 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:57.190985 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e\": container with ID starting with f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e not found: ID does not exist" containerID="f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e" Oct 29 22:41:57.191507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.191016 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e"} err="failed to get container status \"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e\": rpc error: code = NotFound desc = could not find container \"f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e\": container with ID starting with f934dd4ce103280250de514b6f10ba0a08858afdc52d7693492927a548e1743e not found: ID does not exist" Oct 29 22:41:57.226367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.226307 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:41:57.231444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:57.231424 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-675bf4-openshift-default-68f547d6f7-vx5wf"] Oct 29 22:41:57.960836 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:57.960801 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:41:58.387802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:58.387732 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f04497-2263-427b-8521-d79e20303f98" path="/var/lib/kubelet/pods/74f04497-2263-427b-8521-d79e20303f98/volumes" Oct 29 22:41:58.874212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:58.874188 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:41:58.908907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:58.908885 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:41:59.019378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.019352 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\") pod \"03fac3a7-d247-4585-b83f-4bae00efdb4b\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " Oct 29 22:41:59.019550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.019440 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\" (UniqueName: \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\") pod \"864b9a10-f831-472b-bd09-390d86d11f6a\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " Oct 29 22:41:59.019597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.019559 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\" (UniqueName: \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\") pod \"03fac3a7-d247-4585-b83f-4bae00efdb4b\" (UID: \"03fac3a7-d247-4585-b83f-4bae00efdb4b\") " Oct 29 22:41:59.019695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.019677 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\") pod \"864b9a10-f831-472b-bd09-390d86d11f6a\" (UID: \"864b9a10-f831-472b-bd09-390d86d11f6a\") " Oct 29 22:41:59.019789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.019765 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume" pod "03fac3a7-d247-4585-b83f-4bae00efdb4b" (UID: "03fac3a7-d247-4585-b83f-4bae00efdb4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:59.020054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.020028 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume" pod "864b9a10-f831-472b-bd09-390d86d11f6a" (UID: "864b9a10-f831-472b-bd09-390d86d11f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:59.020288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.020271 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864b9a10-f831-472b-bd09-390d86d11f6a-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:59.020349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.020291 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fac3a7-d247-4585-b83f-4bae00efdb4b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:59.021679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.021663 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t" pod "864b9a10-f831-472b-bd09-390d86d11f6a" (UID: "864b9a10-f831-472b-bd09-390d86d11f6a"). InnerVolumeSpecName "kube-api-access-gds8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:59.021753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.021708 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46" pod "03fac3a7-d247-4585-b83f-4bae00efdb4b" (UID: "03fac3a7-d247-4585-b83f-4bae00efdb4b"). InnerVolumeSpecName "kube-api-access-sqw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:59.121500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.121473 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqw46\" (UniqueName: \"kubernetes.io/projected/03fac3a7-d247-4585-b83f-4bae00efdb4b-kube-api-access-sqw46\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:59.121500 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.121498 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gds8t\" (UniqueName: \"kubernetes.io/projected/864b9a10-f831-472b-bd09-390d86d11f6a-kube-api-access-gds8t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:41:59.182363 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.182338 4543 generic.go:356] "Generic (PLEG): container finished" podID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerID="d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7" exitCode=0 Oct 29 22:41:59.182487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.182397 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" Oct 29 22:41:59.182487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.182420 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" event={"ID":"03fac3a7-d247-4585-b83f-4bae00efdb4b","Type":"ContainerDied","Data":"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7"} Oct 29 22:41:59.182487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.182471 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9" event={"ID":"03fac3a7-d247-4585-b83f-4bae00efdb4b","Type":"ContainerDied","Data":"54ed2a9177c6168d4b446168c38e189608b0acb4c8d0b6c90d7a5e1c96c6730c"} Oct 29 22:41:59.182686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.182494 4543 scope.go:122] "RemoveContainer" containerID="d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7" Oct 29 22:41:59.184153 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.184062 4543 generic.go:356] "Generic (PLEG): container finished" podID="864b9a10-f831-472b-bd09-390d86d11f6a" containerID="c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a" exitCode=0 Oct 29 22:41:59.184292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.184278 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" Oct 29 22:41:59.184356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.184295 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" event={"ID":"864b9a10-f831-472b-bd09-390d86d11f6a","Type":"ContainerDied","Data":"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a"} Oct 29 22:41:59.184356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.184343 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn" event={"ID":"864b9a10-f831-472b-bd09-390d86d11f6a","Type":"ContainerDied","Data":"d36c0dc25bd01a1e9ed45103a2c7e94a51762bd20e92883ac27b10293886ec76"} Oct 29 22:41:59.195489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.195470 4543 scope.go:122] "RemoveContainer" containerID="d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7" Oct 29 22:41:59.195747 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:59.195728 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7\": container with ID starting with d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7 not found: ID does not exist" containerID="d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7" Oct 29 22:41:59.195799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.195755 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7"} err="failed to get container status \"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7\": rpc error: code = NotFound desc = could not find container \"d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7\": container with ID starting with d3d4b728a389bb30fe51a7f1e33f95a6493cf235186a282ceb67ad64d9bfcfd7 not found: ID does not exist" Oct 29 22:41:59.195799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.195772 4543 scope.go:122] "RemoveContainer" containerID="c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a" Oct 29 22:41:59.205754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.205736 4543 scope.go:122] "RemoveContainer" containerID="c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a" Oct 29 22:41:59.205997 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:41:59.205977 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a\": container with ID starting with c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a not found: ID does not exist" containerID="c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a" Oct 29 22:41:59.206061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.206003 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a"} err="failed to get container status \"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a\": rpc error: code = NotFound desc = could not find container \"c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a\": container with ID starting with c6b240453f77503eacb806d5fa165264e93f93fd8f29e4376c6b0896ee65c87a not found: ID does not exist" Oct 29 22:41:59.252996 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.252967 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:41:59.266930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.266901 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-99e420-9b7d7758d-mcdrn"] Oct 29 22:41:59.304879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.304824 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:41:59.313033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:41:59.313012 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-99e420-84c87b47c9-mfjs9"] Oct 29 22:42:00.377504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:00.377468 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 22:42:00.387136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:00.387107 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" path="/var/lib/kubelet/pods/03fac3a7-d247-4585-b83f-4bae00efdb4b/volumes" Oct 29 22:42:00.387528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:00.387513 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" path="/var/lib/kubelet/pods/864b9a10-f831-472b-bd09-390d86d11f6a/volumes" Oct 29 22:42:00.420060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:00.420038 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 22:42:02.825906 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:02.825867 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 22:42:02.912243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:02.911615 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 22:42:05.378232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:05.378199 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" containerID="cri-o://92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9" gracePeriod=25 Oct 29 22:42:05.421220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:05.421175 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" containerID="cri-o://db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9" gracePeriod=25 Oct 29 22:42:07.827054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:07.827017 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" containerID="cri-o://1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171" gracePeriod=25 Oct 29 22:42:07.912449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:07.912396 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" containerID="cri-o://9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa" gracePeriod=25 Oct 29 22:42:08.365518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:08.365483 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:08.365709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:08.365486 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:09.688817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:09.688749 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" containerID="cri-o://f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18" gracePeriod=45 Oct 29 22:42:09.996593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:09.996568 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:42:10.031450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031424 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.031598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031576 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\" (UniqueName: \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.031680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.031771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031758 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.031829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031806 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.031902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.031961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.031920 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\") pod \"cc0bac31-3a74-47eb-9106-8096a4440008\" (UID: \"cc0bac31-3a74-47eb-9106-8096a4440008\") " Oct 29 22:42:10.032123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.032066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.032270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.032250 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.032565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.032549 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.032648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.032567 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.032648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.032592 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.033951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.033928 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.034041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.033998 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "kube-api-access-2j9nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:10.034225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.034206 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs" pod "cc0bac31-3a74-47eb-9106-8096a4440008" (UID: "cc0bac31-3a74-47eb-9106-8096a4440008"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:10.134769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.134731 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0bac31-3a74-47eb-9106-8096a4440008-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.134769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.134771 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j9nq\" (UniqueName: \"kubernetes.io/projected/cc0bac31-3a74-47eb-9106-8096a4440008-kube-api-access-2j9nq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.135015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.134788 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0bac31-3a74-47eb-9106-8096a4440008-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.309259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.309169 4543 generic.go:356] "Generic (PLEG): container finished" podID="cc0bac31-3a74-47eb-9106-8096a4440008" containerID="f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18" exitCode=0 Oct 29 22:42:10.309259 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.309230 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" Oct 29 22:42:10.309444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.309256 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" event={"ID":"cc0bac31-3a74-47eb-9106-8096a4440008","Type":"ContainerDied","Data":"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18"} Oct 29 22:42:10.309444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.309303 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" event={"ID":"cc0bac31-3a74-47eb-9106-8096a4440008","Type":"ContainerDied","Data":"f07cb4b0008c05f850ff5a5f80a6c9af0c31794021aae494741f056d5fd616cb"} Oct 29 22:42:10.309444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.309320 4543 scope.go:122] "RemoveContainer" containerID="f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18" Oct 29 22:42:10.321845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.321828 4543 scope.go:122] "RemoveContainer" containerID="f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18" Oct 29 22:42:10.322146 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:10.322126 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18\": container with ID starting with f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18 not found: ID does not exist" containerID="f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18" Oct 29 22:42:10.322232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.322155 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18"} err="failed to get container status \"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18\": rpc error: code = NotFound desc = could not find container \"f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18\": container with ID starting with f9ed3ed43174fcc8e46260a4c539e4450cc4cf5847f87cf9f321f74a7dd46f18 not found: ID does not exist" Oct 29 22:42:10.375247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.375227 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 22:42:10.387725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.387700 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw"] Oct 29 22:42:10.982583 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:10.982543 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/distinct-w1-29c2f2-kserve-7f859bbf88-twnxw" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:42:12.030036 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.030002 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" containerID="cri-o://fbf1732006c310adfef15fdf46ec58b7e57734a51eba9a4ea56af8acf778fb6d" gracePeriod=45 Oct 29 22:42:12.058136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.058104 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" containerID="cri-o://7934379ed9949e09f9aa1f0833354e9e4ad3e1b4421191f11bf7b42e78427830" gracePeriod=45 Oct 29 22:42:12.330513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.330484 4543 generic.go:356] "Generic (PLEG): container finished" podID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerID="fbf1732006c310adfef15fdf46ec58b7e57734a51eba9a4ea56af8acf778fb6d" exitCode=0 Oct 29 22:42:12.330665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.330547 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" event={"ID":"1944fc6c-9357-4c09-af34-46839f8ac52a","Type":"ContainerDied","Data":"fbf1732006c310adfef15fdf46ec58b7e57734a51eba9a4ea56af8acf778fb6d"} Oct 29 22:42:12.330665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.330599 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" event={"ID":"1944fc6c-9357-4c09-af34-46839f8ac52a","Type":"ContainerDied","Data":"996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6"} Oct 29 22:42:12.330665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.330619 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996e9bd62e34628ee54c19fccda76875ef679d6474df33de8a62647dc46878c6" Oct 29 22:42:12.332382 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.332360 4543 generic.go:356] "Generic (PLEG): container finished" podID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerID="7934379ed9949e09f9aa1f0833354e9e4ad3e1b4421191f11bf7b42e78427830" exitCode=0 Oct 29 22:42:12.332505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.332403 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" event={"ID":"81e49f31-2aa9-4e89-ac45-a8947ba56599","Type":"ContainerDied","Data":"7934379ed9949e09f9aa1f0833354e9e4ad3e1b4421191f11bf7b42e78427830"} Oct 29 22:42:12.378219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.378198 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:42:12.382360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.382318 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:42:12.390894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.390864 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" path="/var/lib/kubelet/pods/cc0bac31-3a74-47eb-9106-8096a4440008/volumes" Oct 29 22:42:12.461321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461290 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.461560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461398 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.461560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461457 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.461560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461491 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461613 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461633 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\" (UniqueName: \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461699 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\" (UniqueName: \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461749 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461794 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461831 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461899 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\" (UniqueName: \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.461956 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\") pod \"1944fc6c-9357-4c09-af34-46839f8ac52a\" (UID: \"1944fc6c-9357-4c09-af34-46839f8ac52a\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462012 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\") pod \"81e49f31-2aa9-4e89-ac45-a8947ba56599\" (UID: \"81e49f31-2aa9-4e89-ac45-a8947ba56599\") " Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462398 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462810 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462911 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462933 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462944 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.462971 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.463187 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.465866 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.466122 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.466801 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "kube-api-access-24hm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:12.468628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.467778 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2" pod "81e49f31-2aa9-4e89-ac45-a8947ba56599" (UID: "81e49f31-2aa9-4e89-ac45-a8947ba56599"). InnerVolumeSpecName "kube-api-access-lb2g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:12.470863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.470449 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.472606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.472578 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs" pod "1944fc6c-9357-4c09-af34-46839f8ac52a" (UID: "1944fc6c-9357-4c09-af34-46839f8ac52a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:12.564710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564683 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e49f31-2aa9-4e89-ac45-a8947ba56599-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564708 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hm7\" (UniqueName: \"kubernetes.io/projected/1944fc6c-9357-4c09-af34-46839f8ac52a-kube-api-access-24hm7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564717 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564734 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1944fc6c-9357-4c09-af34-46839f8ac52a-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564746 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1944fc6c-9357-4c09-af34-46839f8ac52a-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564758 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb2g2\" (UniqueName: \"kubernetes.io/projected/81e49f31-2aa9-4e89-ac45-a8947ba56599-kube-api-access-lb2g2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564765 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.564958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:12.564773 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e49f31-2aa9-4e89-ac45-a8947ba56599-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:13.120109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.116270 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:13.120109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.116675 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:13.357052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.356939 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv" Oct 29 22:42:13.358220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.358183 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" event={"ID":"81e49f31-2aa9-4e89-ac45-a8947ba56599","Type":"ContainerDied","Data":"f7c5284671ee9aecc069ce1b061f89bbc4a8f9ec6121639d64713bbe15465d5b"} Oct 29 22:42:13.358380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.358247 4543 scope.go:122] "RemoveContainer" containerID="7934379ed9949e09f9aa1f0833354e9e4ad3e1b4421191f11bf7b42e78427830" Oct 29 22:42:13.358380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.358373 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8" Oct 29 22:42:13.449458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.449421 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 22:42:13.462333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.462304 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/distinct-w4-cea94a-kserve-7679676bd-g5zlv"] Oct 29 22:42:13.506932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.506908 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 22:42:13.512470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.512448 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/simulator-w4-cea94a-kserve-7dbf444464-h46d8"] Oct 29 22:42:13.693464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.693398 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct"] Oct 29 22:42:13.698568 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:13.698547 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-99e420-7f9ct"] Oct 29 22:42:14.386532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:14.386498 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" path="/var/lib/kubelet/pods/1944fc6c-9357-4c09-af34-46839f8ac52a/volumes" Oct 29 22:42:14.386941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:14.386925 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" path="/var/lib/kubelet/pods/81e49f31-2aa9-4e89-ac45-a8947ba56599/volumes" Oct 29 22:42:14.387383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:14.387367 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec32fb9-f848-4734-a77f-12487b915ac9" path="/var/lib/kubelet/pods/9ec32fb9-f848-4734-a77f-12487b915ac9/volumes" Oct 29 22:42:16.328872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.328837 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg"] Oct 29 22:42:16.329627 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329608 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.329669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329628 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.329669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329649 4543 cpu_manager.go:414] "containerMap: removing container" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" Oct 29 22:42:16.329669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329654 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329680 4543 cpu_manager.go:414] "containerMap: removing container" podUID="75043950-90de-4326-a750-5fc8a71383b0" containerName="revoke-keys" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329685 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75043950-90de-4326-a750-5fc8a71383b0" containerName="revoke-keys" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329693 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329698 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329706 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329711 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329722 4543 cpu_manager.go:414] "containerMap: removing container" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329727 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329738 4543 cpu_manager.go:414] "containerMap: removing container" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329743 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329751 4543 cpu_manager.go:414] "containerMap: removing container" podUID="805efdc8-a578-4094-928c-4a660bab9cb7" containerName="revoke-keys" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329755 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805efdc8-a578-4094-928c-4a660bab9cb7" containerName="revoke-keys" Oct 29 22:42:16.329784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329765 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329798 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329814 4543 cpu_manager.go:414] "containerMap: removing container" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329819 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.329995 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330015 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e49f31-2aa9-4e89-ac45-a8947ba56599" containerName="main" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330035 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="75043950-90de-4326-a750-5fc8a71383b0" containerName="revoke-keys" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330042 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c513c5dc-9256-40cf-a616-ada014206f81" containerName="istio-proxy" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330052 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1944fc6c-9357-4c09-af34-46839f8ac52a" containerName="main" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330064 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fac3a7-d247-4585-b83f-4bae00efdb4b" containerName="payload-pre-processing" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330097 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330106 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="805efdc8-a578-4094-928c-4a660bab9cb7" containerName="revoke-keys" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330115 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0bac31-3a74-47eb-9106-8096a4440008" containerName="main" Oct 29 22:42:16.330248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.330128 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="864b9a10-f831-472b-bd09-390d86d11f6a" containerName="payload-processing" Oct 29 22:42:16.335486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.335468 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.360066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.360044 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg"] Oct 29 22:42:16.367230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.367208 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:16.367385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.367272 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:42:16.444169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.444135 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.444524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.444400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.444524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.444464 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djprc\" (UniqueName: \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.545592 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.545568 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.545707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.545608 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djprc\" (UniqueName: \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.545707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.545660 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.546250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.546229 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.548012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.547995 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.574039 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.574009 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djprc\" (UniqueName: \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\") pod \"maas-api-revoke-sub-65421f67-6xjcg\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.649818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.649757 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:16.958129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:16.958096 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg"] Oct 29 22:42:16.959723 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:16.959694 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcc25e2_f3ea_429c_badf_84888839f629.slice/crio-eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae WatchSource:0}: Error finding container eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae: Status 404 returned error can't find the container with id eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae Oct 29 22:42:17.395021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:17.394990 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" event={"ID":"8bcc25e2-f3ea-429c-badf-84888839f629","Type":"ContainerStarted","Data":"9736bdde70ddb82a3c6dd43fb34bd3b12648be8bf16dea8a577f0b9b6d679d35"} Oct 29 22:42:17.395381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:17.395043 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" event={"ID":"8bcc25e2-f3ea-429c-badf-84888839f629","Type":"ContainerStarted","Data":"eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae"} Oct 29 22:42:17.441270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:17.441225 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" podStartSLOduration=1.44121314 podStartE2EDuration="1.44121314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.439521434 +0000 UTC m=+3599.181452108" watchObservedRunningTime="2026-10-29 22:42:17.44121314 +0000 UTC m=+3599.183143812" Oct 29 22:42:18.366178 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:18.366138 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:18.366358 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:18.366158 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:18.404795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:18.404756 4543 generic.go:356] "Generic (PLEG): container finished" podID="8bcc25e2-f3ea-429c-badf-84888839f629" containerID="9736bdde70ddb82a3c6dd43fb34bd3b12648be8bf16dea8a577f0b9b6d679d35" exitCode=0 Oct 29 22:42:18.405191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:18.404792 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" event={"ID":"8bcc25e2-f3ea-429c-badf-84888839f629","Type":"ContainerDied","Data":"9736bdde70ddb82a3c6dd43fb34bd3b12648be8bf16dea8a577f0b9b6d679d35"} Oct 29 22:42:19.568850 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.568828 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:19.687725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.687647 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\" (UniqueName: \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\") pod \"8bcc25e2-f3ea-429c-badf-84888839f629\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " Oct 29 22:42:19.687885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.687853 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\") pod \"8bcc25e2-f3ea-429c-badf-84888839f629\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " Oct 29 22:42:19.687885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.687876 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\") pod \"8bcc25e2-f3ea-429c-badf-84888839f629\" (UID: \"8bcc25e2-f3ea-429c-badf-84888839f629\") " Oct 29 22:42:19.688236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.688211 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca" pod "8bcc25e2-f3ea-429c-badf-84888839f629" (UID: "8bcc25e2-f3ea-429c-badf-84888839f629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:19.688442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.688425 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:19.690117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.690070 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca" pod "8bcc25e2-f3ea-429c-badf-84888839f629" (UID: "8bcc25e2-f3ea-429c-badf-84888839f629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:19.690262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.690245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc" pod "8bcc25e2-f3ea-429c-badf-84888839f629" (UID: "8bcc25e2-f3ea-429c-badf-84888839f629"). InnerVolumeSpecName "kube-api-access-djprc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:19.789270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.789244 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djprc\" (UniqueName: \"kubernetes.io/projected/8bcc25e2-f3ea-429c-badf-84888839f629-kube-api-access-djprc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:19.789270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:19.789268 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcc25e2-f3ea-429c-badf-84888839f629-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:20.423049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:20.423024 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" Oct 29 22:42:20.423277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:20.423056 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg" event={"ID":"8bcc25e2-f3ea-429c-badf-84888839f629","Type":"ContainerDied","Data":"eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae"} Oct 29 22:42:20.423277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:20.423112 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6790903351f39faadf252fd3b0ca47a109a6b7203045b92ab4e07b3fb195ae" Oct 29 22:42:22.801985 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.801949 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9"] Oct 29 22:42:22.802757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.802738 4543 cpu_manager.go:414] "containerMap: removing container" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:22.802813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.802760 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:22.802813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.802783 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8bcc25e2-f3ea-429c-badf-84888839f629" containerName="revoke-keys" Oct 29 22:42:22.802813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.802790 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcc25e2-f3ea-429c-badf-84888839f629" containerName="revoke-keys" Oct 29 22:42:22.803002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.802989 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f04497-2263-427b-8521-d79e20303f98" containerName="istio-proxy" Oct 29 22:42:22.803043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.803012 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcc25e2-f3ea-429c-badf-84888839f629" containerName="revoke-keys" Oct 29 22:42:22.806978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.806961 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:22.823916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.823896 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:42:22.837369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.837351 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:22.842959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.842935 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9"] Oct 29 22:42:22.950616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.950589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8g9\" (UniqueName: \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:22.950811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.950790 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:22.950875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:22.950849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.052180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.052124 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.052275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.052186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.052275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.052259 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8g9\" (UniqueName: \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.052806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.052786 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.054600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.054583 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.078476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.078443 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8g9\" (UniqueName: \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.081680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.081642 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:23.081795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.081706 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:23.120809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.120788 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:23.440476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.440454 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9"] Oct 29 22:42:23.443615 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:23.443590 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4e6f68_a29a_412f_96d7_d25da970bc96.slice/crio-c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75 WatchSource:0}: Error finding container c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75: Status 404 returned error can't find the container with id c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75 Oct 29 22:42:23.463714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:23.463684 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" event={"ID":"be4e6f68-a29a-412f-96d7-d25da970bc96","Type":"ContainerStarted","Data":"c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75"} Oct 29 22:42:24.473675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:24.473649 4543 generic.go:356] "Generic (PLEG): container finished" podID="be4e6f68-a29a-412f-96d7-d25da970bc96" containerID="9fc17f16b0391c26a1e9aeb0711165a3316df33733eac74b8626d9c6093a32bd" exitCode=0 Oct 29 22:42:24.474025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:24.473697 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" event={"ID":"be4e6f68-a29a-412f-96d7-d25da970bc96","Type":"ContainerDied","Data":"9fc17f16b0391c26a1e9aeb0711165a3316df33733eac74b8626d9c6093a32bd"} Oct 29 22:42:24.765018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:24.764992 4543 scope.go:122] "RemoveContainer" containerID="8b134b95249576031519c70ffa0c4d46ee75a5745e5adc21c1e1bc4a0753f49c" Oct 29 22:42:24.775863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:24.775843 4543 scope.go:122] "RemoveContainer" containerID="fbf1732006c310adfef15fdf46ec58b7e57734a51eba9a4ea56af8acf778fb6d" Oct 29 22:42:25.675805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.675772 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:25.747741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.747714 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 22:42:25.772545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.772523 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 22:42:25.790654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.790633 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\" (UniqueName: \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\") pod \"be4e6f68-a29a-412f-96d7-d25da970bc96\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " Oct 29 22:42:25.790881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.790867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\") pod \"be4e6f68-a29a-412f-96d7-d25da970bc96\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " Oct 29 22:42:25.790923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.790904 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\") pod \"be4e6f68-a29a-412f-96d7-d25da970bc96\" (UID: \"be4e6f68-a29a-412f-96d7-d25da970bc96\") " Oct 29 22:42:25.791290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.791264 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca" pod "be4e6f68-a29a-412f-96d7-d25da970bc96" (UID: "be4e6f68-a29a-412f-96d7-d25da970bc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:25.791579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.791560 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:25.793185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.793158 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9" pod "be4e6f68-a29a-412f-96d7-d25da970bc96" (UID: "be4e6f68-a29a-412f-96d7-d25da970bc96"). InnerVolumeSpecName "kube-api-access-kj8g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:25.793295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.793193 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca" pod "be4e6f68-a29a-412f-96d7-d25da970bc96" (UID: "be4e6f68-a29a-412f-96d7-d25da970bc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:25.902493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.902408 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\" (UniqueName: \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\") pod \"790fef06-9bae-4bec-b1ae-1e67749c1295\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " Oct 29 22:42:25.902668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.902556 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\") pod \"790fef06-9bae-4bec-b1ae-1e67749c1295\" (UID: \"790fef06-9bae-4bec-b1ae-1e67749c1295\") " Oct 29 22:42:25.902915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.902893 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume" pod "790fef06-9bae-4bec-b1ae-1e67749c1295" (UID: "790fef06-9bae-4bec-b1ae-1e67749c1295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:25.903050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.902951 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\" (UniqueName: \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\") pod \"b3f8bfba-f489-4e98-b41a-c97c8802c124\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " Oct 29 22:42:25.903050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.902993 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\") pod \"b3f8bfba-f489-4e98-b41a-c97c8802c124\" (UID: \"b3f8bfba-f489-4e98-b41a-c97c8802c124\") " Oct 29 22:42:25.903589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.903559 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e6f68-a29a-412f-96d7-d25da970bc96-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:25.903589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.903582 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8g9\" (UniqueName: \"kubernetes.io/projected/be4e6f68-a29a-412f-96d7-d25da970bc96-kube-api-access-kj8g9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:25.903589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.903592 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790fef06-9bae-4bec-b1ae-1e67749c1295-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:25.904214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.904187 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume" pod "b3f8bfba-f489-4e98-b41a-c97c8802c124" (UID: "b3f8bfba-f489-4e98-b41a-c97c8802c124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:25.906319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.906294 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq" pod "b3f8bfba-f489-4e98-b41a-c97c8802c124" (UID: "b3f8bfba-f489-4e98-b41a-c97c8802c124"). InnerVolumeSpecName "kube-api-access-ghrgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:25.907160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:25.907140 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw" pod "790fef06-9bae-4bec-b1ae-1e67749c1295" (UID: "790fef06-9bae-4bec-b1ae-1e67749c1295"). InnerVolumeSpecName "kube-api-access-z9frw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:26.009652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.005384 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9frw\" (UniqueName: \"kubernetes.io/projected/790fef06-9bae-4bec-b1ae-1e67749c1295-kube-api-access-z9frw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:26.009652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.005422 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrgq\" (UniqueName: \"kubernetes.io/projected/b3f8bfba-f489-4e98-b41a-c97c8802c124-kube-api-access-ghrgq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:26.009652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.005445 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f8bfba-f489-4e98-b41a-c97c8802c124-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:26.491696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.491664 4543 generic.go:356] "Generic (PLEG): container finished" podID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerID="92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9" exitCode=0 Oct 29 22:42:26.491836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.491721 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" Oct 29 22:42:26.491836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.491757 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" event={"ID":"790fef06-9bae-4bec-b1ae-1e67749c1295","Type":"ContainerDied","Data":"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9"} Oct 29 22:42:26.491836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.491809 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7" event={"ID":"790fef06-9bae-4bec-b1ae-1e67749c1295","Type":"ContainerDied","Data":"b129b1aa2c33ea467db2b7ef76eeffc1a7b8d941aefc306b1145d382d63756e5"} Oct 29 22:42:26.491836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.491832 4543 scope.go:122] "RemoveContainer" containerID="92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9" Oct 29 22:42:26.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.493671 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" Oct 29 22:42:26.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.493687 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9" event={"ID":"be4e6f68-a29a-412f-96d7-d25da970bc96","Type":"ContainerDied","Data":"c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75"} Oct 29 22:42:26.494207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.493725 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7fc7fdca0357f660c4b64b7a48906689e195020c5f002c2eaec5f995238ac75" Oct 29 22:42:26.495541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.495463 4543 generic.go:356] "Generic (PLEG): container finished" podID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerID="db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9" exitCode=0 Oct 29 22:42:26.495541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.495511 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" event={"ID":"b3f8bfba-f489-4e98-b41a-c97c8802c124","Type":"ContainerDied","Data":"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9"} Oct 29 22:42:26.495686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.495559 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" Oct 29 22:42:26.495686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.495565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n" event={"ID":"b3f8bfba-f489-4e98-b41a-c97c8802c124","Type":"ContainerDied","Data":"55ac63efc4aad46394dac54338c209e981c4c640edd90a4fe928bc59f456cb2c"} Oct 29 22:42:26.503940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.503919 4543 scope.go:122] "RemoveContainer" containerID="92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9" Oct 29 22:42:26.504261 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:26.504238 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9\": container with ID starting with 92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9 not found: ID does not exist" containerID="92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9" Oct 29 22:42:26.504339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.504268 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9"} err="failed to get container status \"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9\": rpc error: code = NotFound desc = could not find container \"92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9\": container with ID starting with 92c6cdd8cc971d29a421c15a0a5540f0da88d9c7165eb87dbf60476f9247d0a9 not found: ID does not exist" Oct 29 22:42:26.504339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.504286 4543 scope.go:122] "RemoveContainer" containerID="db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9" Oct 29 22:42:26.513823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.513808 4543 scope.go:122] "RemoveContainer" containerID="db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9" Oct 29 22:42:26.514103 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:26.514068 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9\": container with ID starting with db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9 not found: ID does not exist" containerID="db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9" Oct 29 22:42:26.514155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.514111 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9"} err="failed to get container status \"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9\": rpc error: code = NotFound desc = could not find container \"db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9\": container with ID starting with db41c17852e0373c74e708c6a19a96564f94facad437b0c330f4a7034f7792f9 not found: ID does not exist" Oct 29 22:42:26.553798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.553772 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 22:42:26.565653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.565633 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29c2f2-6bfb5fbdc9-7n22n"] Oct 29 22:42:26.606601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.606578 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 22:42:26.613348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:26.613330 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29c2f2-58dc5bb8b8-cdft7"] Oct 29 22:42:27.871719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.871694 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj"] Oct 29 22:42:27.872473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872456 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" Oct 29 22:42:27.872523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872475 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" Oct 29 22:42:27.872523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872501 4543 cpu_manager.go:414] "containerMap: removing container" podUID="be4e6f68-a29a-412f-96d7-d25da970bc96" containerName="revoke-keys" Oct 29 22:42:27.872523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872506 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4e6f68-a29a-412f-96d7-d25da970bc96" containerName="revoke-keys" Oct 29 22:42:27.872523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872517 4543 cpu_manager.go:414] "containerMap: removing container" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" Oct 29 22:42:27.872523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872522 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" Oct 29 22:42:27.872691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872640 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" containerName="payload-pre-processing" Oct 29 22:42:27.872691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872651 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4e6f68-a29a-412f-96d7-d25da970bc96" containerName="revoke-keys" Oct 29 22:42:27.872691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.872657 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" containerName="payload-processing" Oct 29 22:42:27.876324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.876308 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:27.895426 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.895402 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj"] Oct 29 22:42:27.904190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.904144 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:27.904313 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.904267 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:42:27.935971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.935946 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rjq\" (UniqueName: \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:27.936136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.936040 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:27.936136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:27.936123 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:27.964540 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:27.964509 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:42:28.037270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.037241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rjq\" (UniqueName: \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.037396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.037359 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.037454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.037431 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.038018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.037993 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.039789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.039769 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.071032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.070989 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rjq\" (UniqueName: \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\") pod \"maas-api-revoke-sub-76cd7c7c-h6gpj\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.177048 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.177020 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 22:42:28.219053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.219027 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 22:42:28.221786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.221760 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:28.240270 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.240243 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\") pod \"b6a68a24-0c44-4d46-83b4-2469ca64d338\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " Oct 29 22:42:28.240384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.240288 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\" (UniqueName: \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\") pod \"b6a68a24-0c44-4d46-83b4-2469ca64d338\" (UID: \"b6a68a24-0c44-4d46-83b4-2469ca64d338\") " Oct 29 22:42:28.240489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.240470 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\") pod \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " Oct 29 22:42:28.240570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.240555 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\" (UniqueName: \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\") pod \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\" (UID: \"ceddcaa9-b308-4a21-bb62-b92bdf980aa9\") " Oct 29 22:42:28.240637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.240571 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume" pod "b6a68a24-0c44-4d46-83b4-2469ca64d338" (UID: "b6a68a24-0c44-4d46-83b4-2469ca64d338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:28.241172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.241139 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume" pod "ceddcaa9-b308-4a21-bb62-b92bdf980aa9" (UID: "ceddcaa9-b308-4a21-bb62-b92bdf980aa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:28.241279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.241230 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a68a24-0c44-4d46-83b4-2469ca64d338-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:28.243011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.242980 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz" pod "b6a68a24-0c44-4d46-83b4-2469ca64d338" (UID: "b6a68a24-0c44-4d46-83b4-2469ca64d338"). InnerVolumeSpecName "kube-api-access-m7sqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:28.243303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.243279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9" pod "ceddcaa9-b308-4a21-bb62-b92bdf980aa9" (UID: "ceddcaa9-b308-4a21-bb62-b92bdf980aa9"). InnerVolumeSpecName "kube-api-access-4p4m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:28.342866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.342815 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7sqz\" (UniqueName: \"kubernetes.io/projected/b6a68a24-0c44-4d46-83b4-2469ca64d338-kube-api-access-m7sqz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:28.342866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.342851 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:28.342866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.342868 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p4m9\" (UniqueName: \"kubernetes.io/projected/ceddcaa9-b308-4a21-bb62-b92bdf980aa9-kube-api-access-4p4m9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:28.389068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.389002 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790fef06-9bae-4bec-b1ae-1e67749c1295" path="/var/lib/kubelet/pods/790fef06-9bae-4bec-b1ae-1e67749c1295/volumes" Oct 29 22:42:28.389516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.389499 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f8bfba-f489-4e98-b41a-c97c8802c124" path="/var/lib/kubelet/pods/b3f8bfba-f489-4e98-b41a-c97c8802c124/volumes" Oct 29 22:42:28.516043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.516017 4543 generic.go:356] "Generic (PLEG): container finished" podID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerID="9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa" exitCode=0 Oct 29 22:42:28.516182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.516065 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" Oct 29 22:42:28.516182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.516102 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" event={"ID":"ceddcaa9-b308-4a21-bb62-b92bdf980aa9","Type":"ContainerDied","Data":"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa"} Oct 29 22:42:28.516182 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.516157 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk" event={"ID":"ceddcaa9-b308-4a21-bb62-b92bdf980aa9","Type":"ContainerDied","Data":"597cbded6572ef4bde73ecbabe962e7a233fd0182a218ec4bc50a112ae03ba5a"} Oct 29 22:42:28.516339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.516186 4543 scope.go:122] "RemoveContainer" containerID="9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa" Oct 29 22:42:28.517774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.517754 4543 generic.go:356] "Generic (PLEG): container finished" podID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerID="1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171" exitCode=0 Oct 29 22:42:28.517869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.517801 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" Oct 29 22:42:28.517869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.517843 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" event={"ID":"b6a68a24-0c44-4d46-83b4-2469ca64d338","Type":"ContainerDied","Data":"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171"} Oct 29 22:42:28.517978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.517886 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5" event={"ID":"b6a68a24-0c44-4d46-83b4-2469ca64d338","Type":"ContainerDied","Data":"9e4f7cac29c3db8038954b825b4eaa4b11b89401c6c475e8af7ba3bb1ecffaaf"} Oct 29 22:42:28.527716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.527699 4543 scope.go:122] "RemoveContainer" containerID="9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa" Oct 29 22:42:28.528003 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:28.527983 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa\": container with ID starting with 9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa not found: ID does not exist" containerID="9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa" Oct 29 22:42:28.528045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.528010 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa"} err="failed to get container status \"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa\": rpc error: code = NotFound desc = could not find container \"9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa\": container with ID starting with 9e48b3ccf96aaa9b3be2affabf2d91966da2d81e0af85435b115524a1e62c6aa not found: ID does not exist" Oct 29 22:42:28.528045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.528040 4543 scope.go:122] "RemoveContainer" containerID="1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171" Oct 29 22:42:28.536030 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.536006 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj"] Oct 29 22:42:28.538953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.538934 4543 scope.go:122] "RemoveContainer" containerID="1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171" Oct 29 22:42:28.539264 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:28.539240 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171\": container with ID starting with 1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171 not found: ID does not exist" containerID="1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171" Oct 29 22:42:28.539360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.539266 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171"} err="failed to get container status \"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171\": rpc error: code = NotFound desc = could not find container \"1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171\": container with ID starting with 1354a089ba30bb1bd57dd251f322823ac586c99f80a7fcc35b02a69652b7f171 not found: ID does not exist" Oct 29 22:42:28.540799 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:28.540780 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21e98d5_42fd_4e1e_885e_0050da2409c4.slice/crio-cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d WatchSource:0}: Error finding container cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d: Status 404 returned error can't find the container with id cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d Oct 29 22:42:28.576068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.576044 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 22:42:28.583959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.583938 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cea94a-7bc7cbb5b4-fn9p5"] Oct 29 22:42:28.624584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.624554 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 22:42:28.633416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:28.633395 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cea94a-549477b6d7-6tcsk"] Oct 29 22:42:29.573517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:29.573481 4543 generic.go:356] "Generic (PLEG): container finished" podID="b21e98d5-42fd-4e1e-885e-0050da2409c4" containerID="5e6a0e708b956a9f658a1c6e8eb2940b189e4f3635f8f2bd4ca844d7a02224e8" exitCode=0 Oct 29 22:42:29.573986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:29.573567 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" event={"ID":"b21e98d5-42fd-4e1e-885e-0050da2409c4","Type":"ContainerDied","Data":"5e6a0e708b956a9f658a1c6e8eb2940b189e4f3635f8f2bd4ca844d7a02224e8"} Oct 29 22:42:29.573986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:29.573671 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" event={"ID":"b21e98d5-42fd-4e1e-885e-0050da2409c4","Type":"ContainerStarted","Data":"cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d"} Oct 29 22:42:30.386598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.386563 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" path="/var/lib/kubelet/pods/b6a68a24-0c44-4d46-83b4-2469ca64d338/volumes" Oct 29 22:42:30.387043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.387020 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" path="/var/lib/kubelet/pods/ceddcaa9-b308-4a21-bb62-b92bdf980aa9/volumes" Oct 29 22:42:30.744305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.744279 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:30.773318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.773294 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\" (UniqueName: \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\") pod \"b21e98d5-42fd-4e1e-885e-0050da2409c4\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " Oct 29 22:42:30.773450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.773360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\") pod \"b21e98d5-42fd-4e1e-885e-0050da2409c4\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " Oct 29 22:42:30.773522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.773491 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\") pod \"b21e98d5-42fd-4e1e-885e-0050da2409c4\" (UID: \"b21e98d5-42fd-4e1e-885e-0050da2409c4\") " Oct 29 22:42:30.773811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.773788 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca" pod "b21e98d5-42fd-4e1e-885e-0050da2409c4" (UID: "b21e98d5-42fd-4e1e-885e-0050da2409c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:30.774053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.773948 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:30.775498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.775470 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca" pod "b21e98d5-42fd-4e1e-885e-0050da2409c4" (UID: "b21e98d5-42fd-4e1e-885e-0050da2409c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:30.775580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.775558 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq" pod "b21e98d5-42fd-4e1e-885e-0050da2409c4" (UID: "b21e98d5-42fd-4e1e-885e-0050da2409c4"). InnerVolumeSpecName "kube-api-access-j8rjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:30.874825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.874795 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b21e98d5-42fd-4e1e-885e-0050da2409c4-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:30.874825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:30.874822 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rjq\" (UniqueName: \"kubernetes.io/projected/b21e98d5-42fd-4e1e-885e-0050da2409c4-kube-api-access-j8rjq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:31.593482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:31.593444 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" Oct 29 22:42:31.593687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:31.593482 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj" event={"ID":"b21e98d5-42fd-4e1e-885e-0050da2409c4","Type":"ContainerDied","Data":"cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d"} Oct 29 22:42:31.593687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:31.593552 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd176b66b4739513fd1859c2a1f93eea7270619703354a37591902b85170851d" Oct 29 22:42:33.324114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:33.324063 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:42:33.356956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:33.356925 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:42:34.888787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.888733 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk"] Oct 29 22:42:34.889654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889627 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b21e98d5-42fd-4e1e-885e-0050da2409c4" containerName="revoke-keys" Oct 29 22:42:34.889654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889647 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21e98d5-42fd-4e1e-885e-0050da2409c4" containerName="revoke-keys" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889699 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889708 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889720 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889725 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889827 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a68a24-0c44-4d46-83b4-2469ca64d338" containerName="payload-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889839 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceddcaa9-b308-4a21-bb62-b92bdf980aa9" containerName="payload-pre-processing" Oct 29 22:42:34.889862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.889848 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21e98d5-42fd-4e1e-885e-0050da2409c4" containerName="revoke-keys" Oct 29 22:42:34.893752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.893734 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv"] Oct 29 22:42:34.893873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.893852 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:34.898057 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.898038 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:34.905668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.905643 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk"] Oct 29 22:42:34.941626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.941600 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv"] Oct 29 22:42:34.944312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.944285 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:42:34.944418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.944382 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:42:34.963441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963418 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84vh\" (UniqueName: \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:34.963586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963563 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:34.963714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963698 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:34.963758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963727 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxbc\" (UniqueName: \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:34.963807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963771 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:34.963867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:34.963849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.065395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065371 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.065547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065405 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxbc\" (UniqueName: \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.065547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.065547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.065547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065524 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84vh\" (UniqueName: \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.065591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.066307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.066245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.066307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.066253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.067888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.067867 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.068012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.067993 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.091429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.091409 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxbc\" (UniqueName: \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\") pod \"maas-api-revoke-sub-a9b2e232-kbftv\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.091716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.091698 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84vh\" (UniqueName: \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\") pod \"maas-api-revoke-sub-064d209c-btrkk\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.244099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.240686 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:35.244099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.240981 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:35.665297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.665269 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk"] Oct 29 22:42:35.666377 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:35.666344 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3618d3_1f3e_483e_bfbf_6c1b1a3072e3.slice/crio-49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8 WatchSource:0}: Error finding container 49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8: Status 404 returned error can't find the container with id 49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8 Oct 29 22:42:35.690121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:35.690060 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv"] Oct 29 22:42:35.702505 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:35.702437 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14367e8f_e349_46dc_9535_2822050b29e0.slice/crio-ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049 WatchSource:0}: Error finding container ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049: Status 404 returned error can't find the container with id ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049 Oct 29 22:42:36.659872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.659837 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" event={"ID":"14367e8f-e349-46dc-9535-2822050b29e0","Type":"ContainerStarted","Data":"5058159716c7b1167636fa1129d25c67e37211afb70f351ee838859feacd57c9"} Oct 29 22:42:36.660309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.659898 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" event={"ID":"14367e8f-e349-46dc-9535-2822050b29e0","Type":"ContainerStarted","Data":"ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049"} Oct 29 22:42:36.661506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.661480 4543 generic.go:356] "Generic (PLEG): container finished" podID="1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" containerID="0e88eb314942f506d4bd614ed6295a92284d402e5568d0c75e7387fc5b986eae" exitCode=0 Oct 29 22:42:36.661621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.661540 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" event={"ID":"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3","Type":"ContainerDied","Data":"0e88eb314942f506d4bd614ed6295a92284d402e5568d0c75e7387fc5b986eae"} Oct 29 22:42:36.661621 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.661582 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" event={"ID":"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3","Type":"ContainerStarted","Data":"49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8"} Oct 29 22:42:36.710632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:36.710589 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" podStartSLOduration=2.710575462 podStartE2EDuration="2.710575462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:36.706287326 +0000 UTC m=+3618.448217998" watchObservedRunningTime="2026-10-29 22:42:36.710575462 +0000 UTC m=+3618.452506132" Oct 29 22:42:37.671271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.671238 4543 generic.go:356] "Generic (PLEG): container finished" podID="14367e8f-e349-46dc-9535-2822050b29e0" containerID="5058159716c7b1167636fa1129d25c67e37211afb70f351ee838859feacd57c9" exitCode=0 Oct 29 22:42:37.671693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.671322 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" event={"ID":"14367e8f-e349-46dc-9535-2822050b29e0","Type":"ContainerDied","Data":"5058159716c7b1167636fa1129d25c67e37211afb70f351ee838859feacd57c9"} Oct 29 22:42:37.836558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.836529 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:37.919115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.919089 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\") pod \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " Oct 29 22:42:37.919237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.919172 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\" (UniqueName: \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\") pod \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " Oct 29 22:42:37.919237 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.919207 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\") pod \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\" (UID: \"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3\") " Oct 29 22:42:37.919450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.919422 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca" pod "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" (UID: "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:37.919599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.919584 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:37.921414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.921362 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh" pod "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" (UID: "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3"). InnerVolumeSpecName "kube-api-access-m84vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:37.921414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:37.921406 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca" pod "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" (UID: "1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:38.021084 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.021055 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84vh\" (UniqueName: \"kubernetes.io/projected/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-kube-api-access-m84vh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:38.021197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.021108 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:38.324801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.324761 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" containerID="cri-o://67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9" gracePeriod=25 Oct 29 22:42:38.357609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.357582 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" containerID="cri-o://e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b" gracePeriod=25 Oct 29 22:42:38.643364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.643303 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx"] Oct 29 22:42:38.644008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.643994 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" containerName="revoke-keys" Oct 29 22:42:38.644059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.644010 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" containerName="revoke-keys" Oct 29 22:42:38.644157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.644146 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" containerName="revoke-keys" Oct 29 22:42:38.647971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.647953 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.661387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.661364 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb"] Oct 29 22:42:38.666299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.666279 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.667576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.667556 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx"] Oct 29 22:42:38.688388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.688364 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" Oct 29 22:42:38.692611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.688606 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk" event={"ID":"1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3","Type":"ContainerDied","Data":"49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8"} Oct 29 22:42:38.692611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.690846 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b95b389927a599683b91945b591098226d4308d89bb1718c4e5f4c533c4bf8" Oct 29 22:42:38.692611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.690877 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb"] Oct 29 22:42:38.728654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.728626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwddp\" (UniqueName: \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.728810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.728681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.728810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.728766 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.728980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.728901 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.728980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.728967 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.729141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.729065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtsd\" (UniqueName: \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.830015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.829984 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.830225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830022 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.830225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830092 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.830225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830127 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.830225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830168 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtsd\" (UniqueName: \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.831333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830321 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwddp\" (UniqueName: \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.831333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.830952 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.831333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.831167 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.833121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.833096 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.833279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.833259 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.856836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.856805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtsd\" (UniqueName: \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\") pod \"maas-api-revoke-sub-8c005b17-ddwvx\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.856965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.856813 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwddp\" (UniqueName: \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\") pod \"maas-api-revoke-sub-ee52575a-sbzgb\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:38.860051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.860027 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:38.931125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.931024 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\") pod \"14367e8f-e349-46dc-9535-2822050b29e0\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " Oct 29 22:42:38.931125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.931109 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\" (UniqueName: \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\") pod \"14367e8f-e349-46dc-9535-2822050b29e0\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " Oct 29 22:42:38.931367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.931199 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\") pod \"14367e8f-e349-46dc-9535-2822050b29e0\" (UID: \"14367e8f-e349-46dc-9535-2822050b29e0\") " Oct 29 22:42:38.931898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.931864 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca" pod "14367e8f-e349-46dc-9535-2822050b29e0" (UID: "14367e8f-e349-46dc-9535-2822050b29e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:38.937199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.937171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca" pod "14367e8f-e349-46dc-9535-2822050b29e0" (UID: "14367e8f-e349-46dc-9535-2822050b29e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:38.941958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.941928 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc" pod "14367e8f-e349-46dc-9535-2822050b29e0" (UID: "14367e8f-e349-46dc-9535-2822050b29e0"). InnerVolumeSpecName "kube-api-access-bkxbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:38.962542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.962520 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:38.983736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:38.983715 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:39.033240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.033208 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxbc\" (UniqueName: \"kubernetes.io/projected/14367e8f-e349-46dc-9535-2822050b29e0-kube-api-access-bkxbc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.033354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.033246 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.033354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.033262 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14367e8f-e349-46dc-9535-2822050b29e0-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:39.318064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.318035 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx"] Oct 29 22:42:39.318851 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:39.318823 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd0b54_6b3c_4a29_b4ff_95db857b425f.slice/crio-98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81 WatchSource:0}: Error finding container 98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81: Status 404 returned error can't find the container with id 98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81 Oct 29 22:42:39.335487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.335467 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb"] Oct 29 22:42:39.338128 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:39.338102 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaece6e8b_90f4_420b_a3cb_ead1ffaf3b3d.slice/crio-0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e WatchSource:0}: Error finding container 0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e: Status 404 returned error can't find the container with id 0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e Oct 29 22:42:39.403337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.403301 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8"] Oct 29 22:42:39.404022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.404008 4543 cpu_manager.go:414] "containerMap: removing container" podUID="14367e8f-e349-46dc-9535-2822050b29e0" containerName="revoke-keys" Oct 29 22:42:39.404110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.404024 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14367e8f-e349-46dc-9535-2822050b29e0" containerName="revoke-keys" Oct 29 22:42:39.404229 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.404217 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="14367e8f-e349-46dc-9535-2822050b29e0" containerName="revoke-keys" Oct 29 22:42:39.409781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.409764 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.507903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.507873 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8"] Oct 29 22:42:39.540753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.540725 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtnp\" (UniqueName: \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.540911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.540893 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.541016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.541003 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.641761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.641737 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.641841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.641813 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.641886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.641870 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtnp\" (UniqueName: \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.642563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.642544 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.644371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.644265 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.675661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.675640 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtnp\" (UniqueName: \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\") pod \"maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.697287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.697259 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" event={"ID":"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d","Type":"ContainerStarted","Data":"0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e"} Oct 29 22:42:39.698952 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.698925 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" event={"ID":"27fd0b54-6b3c-4a29-b4ff-95db857b425f","Type":"ContainerStarted","Data":"cfaad37594b564b17869a27ef19894fbd1eff72656b48bc555ccce06e72e09f5"} Oct 29 22:42:39.699055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.698974 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" event={"ID":"27fd0b54-6b3c-4a29-b4ff-95db857b425f","Type":"ContainerStarted","Data":"98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81"} Oct 29 22:42:39.700403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.700386 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" Oct 29 22:42:39.700481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.700426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv" event={"ID":"14367e8f-e349-46dc-9535-2822050b29e0","Type":"ContainerDied","Data":"ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049"} Oct 29 22:42:39.700481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.700466 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0186e5196749f56fe802ddc77cb48a957a50aa15fd75d3d2f1d19b3c16b049" Oct 29 22:42:39.724251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.724234 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:39.770747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:39.770696 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" podStartSLOduration=1.7706811230000001 podStartE2EDuration="1.770681123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:39.766215192 +0000 UTC m=+3621.508145866" watchObservedRunningTime="2026-10-29 22:42:39.770681123 +0000 UTC m=+3621.512611831" Oct 29 22:42:40.039409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.039380 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8"] Oct 29 22:42:40.039899 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:42:40.039868 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f48b970_7faa_4aae_9a8e_c5363154c415.slice/crio-6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8 WatchSource:0}: Error finding container 6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8: Status 404 returned error can't find the container with id 6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8 Oct 29 22:42:40.714486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.714445 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" event={"ID":"0f48b970-7faa-4aae-9a8e-c5363154c415","Type":"ContainerStarted","Data":"f55f28b06250856faec7f1892785d43d493756ac4377614e27876a0f8a2ab47b"} Oct 29 22:42:40.714880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.714510 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" event={"ID":"0f48b970-7faa-4aae-9a8e-c5363154c415","Type":"ContainerStarted","Data":"6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8"} Oct 29 22:42:40.716117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.716094 4543 generic.go:356] "Generic (PLEG): container finished" podID="aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" containerID="022413909f3e80108241640df4fbb427e886cbf40da3a8e2e294c2e43c2877c8" exitCode=0 Oct 29 22:42:40.716217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.716162 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" event={"ID":"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d","Type":"ContainerDied","Data":"022413909f3e80108241640df4fbb427e886cbf40da3a8e2e294c2e43c2877c8"} Oct 29 22:42:40.717774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.717748 4543 generic.go:356] "Generic (PLEG): container finished" podID="27fd0b54-6b3c-4a29-b4ff-95db857b425f" containerID="cfaad37594b564b17869a27ef19894fbd1eff72656b48bc555ccce06e72e09f5" exitCode=0 Oct 29 22:42:40.717860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.717828 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" event={"ID":"27fd0b54-6b3c-4a29-b4ff-95db857b425f","Type":"ContainerDied","Data":"cfaad37594b564b17869a27ef19894fbd1eff72656b48bc555ccce06e72e09f5"} Oct 29 22:42:40.761369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:40.761319 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" podStartSLOduration=1.761305851 podStartE2EDuration="1.761305851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:40.757890195 +0000 UTC m=+3622.499820865" watchObservedRunningTime="2026-10-29 22:42:40.761305851 +0000 UTC m=+3622.503236547" Oct 29 22:42:41.727551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.727516 4543 generic.go:356] "Generic (PLEG): container finished" podID="0f48b970-7faa-4aae-9a8e-c5363154c415" containerID="f55f28b06250856faec7f1892785d43d493756ac4377614e27876a0f8a2ab47b" exitCode=0 Oct 29 22:42:41.728004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.727614 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" event={"ID":"0f48b970-7faa-4aae-9a8e-c5363154c415","Type":"ContainerDied","Data":"f55f28b06250856faec7f1892785d43d493756ac4377614e27876a0f8a2ab47b"} Oct 29 22:42:41.910730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.910704 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:41.965337 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.965311 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:41.981888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.981820 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\") pod \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " Oct 29 22:42:41.982016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.981941 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\") pod \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " Oct 29 22:42:41.982016 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.982010 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\" (UniqueName: \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\") pod \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\" (UID: \"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d\") " Oct 29 22:42:41.982385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.982360 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca" pod "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" (UID: "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:41.982591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.982572 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:41.984060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.984039 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca" pod "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" (UID: "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:41.984243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:41.984225 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp" pod "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" (UID: "aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d"). InnerVolumeSpecName "kube-api-access-nwddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:42.084026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084001 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\") pod \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " Oct 29 22:42:42.084201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084123 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\" (UniqueName: \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\") pod \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " Oct 29 22:42:42.084201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084146 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\") pod \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\" (UID: \"27fd0b54-6b3c-4a29-b4ff-95db857b425f\") " Oct 29 22:42:42.084563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084540 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca" pod "27fd0b54-6b3c-4a29-b4ff-95db857b425f" (UID: "27fd0b54-6b3c-4a29-b4ff-95db857b425f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:42.084806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084791 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.084858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084808 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.084858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.084817 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwddp\" (UniqueName: \"kubernetes.io/projected/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d-kube-api-access-nwddp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.086197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.086174 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca" pod "27fd0b54-6b3c-4a29-b4ff-95db857b425f" (UID: "27fd0b54-6b3c-4a29-b4ff-95db857b425f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:42.086344 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.086325 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd" pod "27fd0b54-6b3c-4a29-b4ff-95db857b425f" (UID: "27fd0b54-6b3c-4a29-b4ff-95db857b425f"). InnerVolumeSpecName "kube-api-access-wgtsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:42.186508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.186481 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtsd\" (UniqueName: \"kubernetes.io/projected/27fd0b54-6b3c-4a29-b4ff-95db857b425f-kube-api-access-wgtsd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.186508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.186509 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fd0b54-6b3c-4a29-b4ff-95db857b425f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.737322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.737293 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" Oct 29 22:42:42.737717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.737326 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb" event={"ID":"aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d","Type":"ContainerDied","Data":"0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e"} Oct 29 22:42:42.737717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.737375 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d761acfb170f72a479971eaf752c8e5ffc22e09a3da065b51a259a4100c740e" Oct 29 22:42:42.738781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.738760 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" Oct 29 22:42:42.738902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.738805 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx" event={"ID":"27fd0b54-6b3c-4a29-b4ff-95db857b425f","Type":"ContainerDied","Data":"98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81"} Oct 29 22:42:42.738902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.738852 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d9c3db72f3f0426458fd819e783f6dddef16f0b421c9ead5a9a95b495a7c81" Oct 29 22:42:42.900296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:42.900267 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:43.033181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.033069 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\") pod \"0f48b970-7faa-4aae-9a8e-c5363154c415\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " Oct 29 22:42:43.033370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.033241 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\") pod \"0f48b970-7faa-4aae-9a8e-c5363154c415\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " Oct 29 22:42:43.033692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.033662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca" pod "0f48b970-7faa-4aae-9a8e-c5363154c415" (UID: "0f48b970-7faa-4aae-9a8e-c5363154c415"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:43.033821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.033807 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\" (UniqueName: \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\") pod \"0f48b970-7faa-4aae-9a8e-c5363154c415\" (UID: \"0f48b970-7faa-4aae-9a8e-c5363154c415\") " Oct 29 22:42:43.034956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.034510 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:43.035652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.035620 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca" pod "0f48b970-7faa-4aae-9a8e-c5363154c415" (UID: "0f48b970-7faa-4aae-9a8e-c5363154c415"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:43.036316 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.036293 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp" pod "0f48b970-7faa-4aae-9a8e-c5363154c415" (UID: "0f48b970-7faa-4aae-9a8e-c5363154c415"). InnerVolumeSpecName "kube-api-access-nbtnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:43.135682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.135655 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f48b970-7faa-4aae-9a8e-c5363154c415-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:43.135682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.135680 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbtnp\" (UniqueName: \"kubernetes.io/projected/0f48b970-7faa-4aae-9a8e-c5363154c415-kube-api-access-nbtnp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:43.758139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.758109 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" Oct 29 22:42:43.758609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.758109 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8" event={"ID":"0f48b970-7faa-4aae-9a8e-c5363154c415","Type":"ContainerDied","Data":"6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8"} Oct 29 22:42:43.758609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:43.758263 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbb3da2a92b292b6c132c08f111ba3aef86354de0e8026f70e50e8d4ae980d8" Oct 29 22:42:47.190755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:47.190722 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:47.191165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:47.190726 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:49.956418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:49.956385 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:42:54.957301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:54.957260 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" containerID="cri-o://bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462" gracePeriod=25 Oct 29 22:42:56.207419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:56.207383 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:57.190786 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:57.190751 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:57.190959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:57.190795 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:42:57.919875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:57.919838 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9"] Oct 29 22:42:57.932969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:57.932940 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-15154b-b6dj9"] Oct 29 22:42:57.961199 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:57.961170 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:42:58.387115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.387056 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4e6f68-a29a-412f-96d7-d25da970bc96" path="/var/lib/kubelet/pods/be4e6f68-a29a-412f-96d7-d25da970bc96/volumes" Oct 29 22:42:58.641340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.641316 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:42:58.678760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.678737 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:42:58.735830 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.735799 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\") pod \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " Oct 29 22:42:58.735986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.735897 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\" (UniqueName: \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\") pod \"7402b9b5-9920-4800-82d1-d35f3f0b0426\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " Oct 29 22:42:58.735986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.735969 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\" (UniqueName: \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\") pod \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\" (UID: \"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff\") " Oct 29 22:42:58.736251 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.736214 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\") pod \"7402b9b5-9920-4800-82d1-d35f3f0b0426\" (UID: \"7402b9b5-9920-4800-82d1-d35f3f0b0426\") " Oct 29 22:42:58.736902 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.736867 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume" pod "7402b9b5-9920-4800-82d1-d35f3f0b0426" (UID: "7402b9b5-9920-4800-82d1-d35f3f0b0426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:58.737376 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.737355 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume" pod "98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" (UID: "98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:58.739186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.739149 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8" pod "98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" (UID: "98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff"). InnerVolumeSpecName "kube-api-access-d8hq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.744108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.744057 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp" pod "7402b9b5-9920-4800-82d1-d35f3f0b0426" (UID: "7402b9b5-9920-4800-82d1-d35f3f0b0426"). InnerVolumeSpecName "kube-api-access-8l4cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:58.837679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.837653 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7402b9b5-9920-4800-82d1-d35f3f0b0426-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.837679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.837680 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.837847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.837694 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4cp\" (UniqueName: \"kubernetes.io/projected/7402b9b5-9920-4800-82d1-d35f3f0b0426-kube-api-access-8l4cp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.837847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.837706 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8hq8\" (UniqueName: \"kubernetes.io/projected/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff-kube-api-access-d8hq8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:42:58.878678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.878613 4543 generic.go:356] "Generic (PLEG): container finished" podID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerID="e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b" exitCode=0 Oct 29 22:42:58.878678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.878663 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" Oct 29 22:42:58.878847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.878698 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" event={"ID":"7402b9b5-9920-4800-82d1-d35f3f0b0426","Type":"ContainerDied","Data":"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b"} Oct 29 22:42:58.878847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.878750 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp" event={"ID":"7402b9b5-9920-4800-82d1-d35f3f0b0426","Type":"ContainerDied","Data":"e046f3abeacb72cc0799d597d05d48a23ea91bdcc5bce99cb2fde1ae22afedaa"} Oct 29 22:42:58.878847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.878778 4543 scope.go:122] "RemoveContainer" containerID="e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b" Oct 29 22:42:58.880558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.880536 4543 generic.go:356] "Generic (PLEG): container finished" podID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerID="67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9" exitCode=0 Oct 29 22:42:58.880685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.880584 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" event={"ID":"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff","Type":"ContainerDied","Data":"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9"} Oct 29 22:42:58.880685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.880586 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" Oct 29 22:42:58.880685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.880615 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5" event={"ID":"98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff","Type":"ContainerDied","Data":"54496cb6f9fb2e8527685c86728a1f9f8224d73cf0e1915ccd773de8b7b4f4ea"} Oct 29 22:42:58.890238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.890223 4543 scope.go:122] "RemoveContainer" containerID="e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b" Oct 29 22:42:58.890500 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:58.890479 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b\": container with ID starting with e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b not found: ID does not exist" containerID="e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b" Oct 29 22:42:58.890564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.890505 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b"} err="failed to get container status \"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b\": rpc error: code = NotFound desc = could not find container \"e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b\": container with ID starting with e3507cf057a7956c336833fa30be72f61a2e72a450257351cce9edf2d8f6670b not found: ID does not exist" Oct 29 22:42:58.890564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.890521 4543 scope.go:122] "RemoveContainer" containerID="67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9" Oct 29 22:42:58.903556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.903526 4543 scope.go:122] "RemoveContainer" containerID="67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9" Oct 29 22:42:58.903946 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:42:58.903926 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9\": container with ID starting with 67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9 not found: ID does not exist" containerID="67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9" Oct 29 22:42:58.904024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.903952 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9"} err="failed to get container status \"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9\": rpc error: code = NotFound desc = could not find container \"67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9\": container with ID starting with 67d8c4e8752abf9a64a66c4cb423d1f5c29319100edb89f34bc9e039935e97f9 not found: ID does not exist" Oct 29 22:42:58.943238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.943212 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:42:58.948154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.948135 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-15154b-79dddf54f6-p2dr5"] Oct 29 22:42:58.981881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.981857 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:42:58.990361 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:42:58.990341 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-15154b-58949fdf59-27wpp"] Oct 29 22:43:00.386128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:00.386098 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" path="/var/lib/kubelet/pods/7402b9b5-9920-4800-82d1-d35f3f0b0426/volumes" Oct 29 22:43:00.386510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:00.386493 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" path="/var/lib/kubelet/pods/98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff/volumes" Oct 29 22:43:06.207414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:06.207369 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:07.132968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.132935 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b"] Oct 29 22:43:07.133650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133629 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" Oct 29 22:43:07.133650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133646 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133682 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0f48b970-7faa-4aae-9a8e-c5363154c415" containerName="revoke-keys" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133687 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f48b970-7faa-4aae-9a8e-c5363154c415" containerName="revoke-keys" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133702 4543 cpu_manager.go:414] "containerMap: removing container" podUID="27fd0b54-6b3c-4a29-b4ff-95db857b425f" containerName="revoke-keys" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133708 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fd0b54-6b3c-4a29-b4ff-95db857b425f" containerName="revoke-keys" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133727 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133732 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133739 4543 cpu_manager.go:414] "containerMap: removing container" podUID="aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" containerName="revoke-keys" Oct 29 22:43:07.133821 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133744 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" containerName="revoke-keys" Oct 29 22:43:07.134102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133845 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f48b970-7faa-4aae-9a8e-c5363154c415" containerName="revoke-keys" Oct 29 22:43:07.134102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133853 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fd0b54-6b3c-4a29-b4ff-95db857b425f" containerName="revoke-keys" Oct 29 22:43:07.134102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133863 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c6e527-f6e0-4dae-8b3b-2a30cd9c6fff" containerName="payload-processing" Oct 29 22:43:07.134102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133870 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" containerName="revoke-keys" Oct 29 22:43:07.134102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.133877 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7402b9b5-9920-4800-82d1-d35f3f0b0426" containerName="payload-pre-processing" Oct 29 22:43:07.139277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.139260 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.164105 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.164059 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b"] Oct 29 22:43:07.171212 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.171189 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:43:07.171321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.171266 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:43:07.235111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.235085 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbz94\" (UniqueName: \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.235454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.235255 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.235454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.235295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.336876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.336849 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.337004 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.336892 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.337070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.337052 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbz94\" (UniqueName: \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.337636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.337614 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.339439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.339418 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.363271 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.363242 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbz94\" (UniqueName: \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\") pod \"maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.453869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.453821 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:07.764321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.764293 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b"] Oct 29 22:43:07.765425 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:43:07.765400 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddcc861_a779_410c_9d66_89eaeeb8cb0e.slice/crio-71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2 WatchSource:0}: Error finding container 71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2: Status 404 returned error can't find the container with id 71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2 Oct 29 22:43:07.989689 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:07.989660 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" event={"ID":"8ddcc861-a779-410c-9d66-89eaeeb8cb0e","Type":"ContainerStarted","Data":"71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2"} Oct 29 22:43:09.000012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:08.999971 4543 generic.go:356] "Generic (PLEG): container finished" podID="8ddcc861-a779-410c-9d66-89eaeeb8cb0e" containerID="beb39cb7e0059427d884b129e928bcafb80995a64c8014441917769ecb67b6fd" exitCode=0 Oct 29 22:43:09.000495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:09.000016 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" event={"ID":"8ddcc861-a779-410c-9d66-89eaeeb8cb0e","Type":"ContainerDied","Data":"beb39cb7e0059427d884b129e928bcafb80995a64c8014441917769ecb67b6fd"} Oct 29 22:43:10.173207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.173182 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:10.283533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.283465 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\" (UniqueName: \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\") pod \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " Oct 29 22:43:10.283672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.283592 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\") pod \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " Oct 29 22:43:10.283717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.283701 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\") pod \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\" (UID: \"8ddcc861-a779-410c-9d66-89eaeeb8cb0e\") " Oct 29 22:43:10.283974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.283943 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca" pod "8ddcc861-a779-410c-9d66-89eaeeb8cb0e" (UID: "8ddcc861-a779-410c-9d66-89eaeeb8cb0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:10.284140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.284105 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:10.285752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.285733 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94" pod "8ddcc861-a779-410c-9d66-89eaeeb8cb0e" (UID: "8ddcc861-a779-410c-9d66-89eaeeb8cb0e"). InnerVolumeSpecName "kube-api-access-kbz94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:10.285851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.285773 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca" pod "8ddcc861-a779-410c-9d66-89eaeeb8cb0e" (UID: "8ddcc861-a779-410c-9d66-89eaeeb8cb0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:10.384820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.384794 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:10.384820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:10.384818 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbz94\" (UniqueName: \"kubernetes.io/projected/8ddcc861-a779-410c-9d66-89eaeeb8cb0e-kube-api-access-kbz94\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:11.017878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:11.017805 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" Oct 29 22:43:11.017878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:11.017839 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b" event={"ID":"8ddcc861-a779-410c-9d66-89eaeeb8cb0e","Type":"ContainerDied","Data":"71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2"} Oct 29 22:43:11.018146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:11.017887 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71755a0815b2811564de2a08f622fa6c503a353da028b0a97ecceca2cb2ccaa2" Oct 29 22:43:14.696196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:14.696155 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8"] Oct 29 22:43:14.704608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:14.704573 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c06a2c-lc8l8"] Oct 29 22:43:15.238047 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.237980 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:43:15.359648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.359613 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\") pod \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " Oct 29 22:43:15.359849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.359680 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\" (UniqueName: \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\") pod \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\" (UID: \"c51c89c9-aebd-40b1-927f-9dbe7b85db51\") " Oct 29 22:43:15.360331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.360303 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume" pod "c51c89c9-aebd-40b1-927f-9dbe7b85db51" (UID: "c51c89c9-aebd-40b1-927f-9dbe7b85db51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:15.362312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.362283 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg" pod "c51c89c9-aebd-40b1-927f-9dbe7b85db51" (UID: "c51c89c9-aebd-40b1-927f-9dbe7b85db51"). InnerVolumeSpecName "kube-api-access-7k6tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:15.461351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.461329 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51c89c9-aebd-40b1-927f-9dbe7b85db51-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:15.461351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:15.461352 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6tg\" (UniqueName: \"kubernetes.io/projected/c51c89c9-aebd-40b1-927f-9dbe7b85db51-kube-api-access-7k6tg\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:16.085524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.085486 4543 generic.go:356] "Generic (PLEG): container finished" podID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerID="bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462" exitCode=0 Oct 29 22:43:16.086062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.085543 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" Oct 29 22:43:16.086062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.085566 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" event={"ID":"c51c89c9-aebd-40b1-927f-9dbe7b85db51","Type":"ContainerDied","Data":"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462"} Oct 29 22:43:16.086062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.085658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh" event={"ID":"c51c89c9-aebd-40b1-927f-9dbe7b85db51","Type":"ContainerDied","Data":"40799d57093bc639f39978aa1b58b0c6cc6cba4268a8549090ede6429d61016e"} Oct 29 22:43:16.086062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.085687 4543 scope.go:122] "RemoveContainer" containerID="bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462" Oct 29 22:43:16.101247 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.101183 4543 scope.go:122] "RemoveContainer" containerID="bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462" Oct 29 22:43:16.101535 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:43:16.101507 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462\": container with ID starting with bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462 not found: ID does not exist" containerID="bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462" Oct 29 22:43:16.101637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.101546 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462"} err="failed to get container status \"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462\": rpc error: code = NotFound desc = could not find container \"bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462\": container with ID starting with bcba0f110c35046c6f34c49afab12d4100eeee4d5ba93b52eab1888815b56462 not found: ID does not exist" Oct 29 22:43:16.149397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.149370 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:43:16.162107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.162070 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c06a2c-97f78fbf8-p8hmh"] Oct 29 22:43:16.396389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.396294 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f48b970-7faa-4aae-9a8e-c5363154c415" path="/var/lib/kubelet/pods/0f48b970-7faa-4aae-9a8e-c5363154c415/volumes" Oct 29 22:43:16.396890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.396870 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" path="/var/lib/kubelet/pods/c51c89c9-aebd-40b1-927f-9dbe7b85db51/volumes" Oct 29 22:43:16.840739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.840704 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v"] Oct 29 22:43:16.846553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:16.846515 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-29c2f2-7ls5v"] Oct 29 22:43:17.613989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:17.613955 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 22:43:17.663027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:17.662995 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 22:43:18.515145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:18.513524 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75043950-90de-4326-a750-5fc8a71383b0" path="/var/lib/kubelet/pods/75043950-90de-4326-a750-5fc8a71383b0/volumes" Oct 29 22:43:19.203872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:19.203833 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7"] Oct 29 22:43:19.217374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:19.217339 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cea94a-zgkw7"] Oct 29 22:43:20.386908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:20.386876 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805efdc8-a578-4094-928c-4a660bab9cb7" path="/var/lib/kubelet/pods/805efdc8-a578-4094-928c-4a660bab9cb7/volumes" Oct 29 22:43:22.615042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:22.614992 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" containerID="cri-o://07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708" gracePeriod=25 Oct 29 22:43:22.664126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:22.664061 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" containerID="cri-o://33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd" gracePeriod=25 Oct 29 22:43:23.010288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.010251 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:43:23.010794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.010755 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" containerID="cri-o://9448e0f1c921c054ec3b61c5aee3ee594a78969379654cac4f895ad5f4e5af63" gracePeriod=30 Oct 29 22:43:23.081872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.081838 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:23.577749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.577717 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr"] Oct 29 22:43:23.578427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578413 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8ddcc861-a779-410c-9d66-89eaeeb8cb0e" containerName="revoke-keys" Oct 29 22:43:23.578493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578428 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ddcc861-a779-410c-9d66-89eaeeb8cb0e" containerName="revoke-keys" Oct 29 22:43:23.578493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578462 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" Oct 29 22:43:23.578493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578467 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" Oct 29 22:43:23.578605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578562 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c89c9-aebd-40b1-927f-9dbe7b85db51" containerName="payload-processing" Oct 29 22:43:23.578605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.578577 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ddcc861-a779-410c-9d66-89eaeeb8cb0e" containerName="revoke-keys" Oct 29 22:43:23.581981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.581965 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.603981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.603957 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr"] Oct 29 22:43:23.612125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.612104 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:43:23.612326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.612305 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:43:23.636391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.636359 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.636792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.636407 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7tj\" (UniqueName: \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.636792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.636590 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.737792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.737758 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.737919 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.737881 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.737995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.737927 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7tj\" (UniqueName: \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.738534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.738467 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.740171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.740153 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.763988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.763964 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7tj\" (UniqueName: \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\") pod \"maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:23.896341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:23.896278 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:24.213314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:24.213283 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr"] Oct 29 22:43:24.214293 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:43:24.214265 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025ca3d2_dc28_4d52_8dc7_80869db6e538.slice/crio-8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e WatchSource:0}: Error finding container 8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e: Status 404 returned error can't find the container with id 8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e Oct 29 22:43:24.544734 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:24.544708 4543 patch_prober.go:29] interesting pod/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:43:24.544859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:24.544763 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:43:24.857049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:24.856976 4543 scope.go:122] "RemoveContainer" containerID="a870ec98b2249939f2acab1a954d99289e0e4585c7bdcf4221c9be4e1fd1d0e1" Oct 29 22:43:24.867616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:24.867594 4543 scope.go:122] "RemoveContainer" containerID="bd2b412f55b05a669abef97447fe0e288b0570b59a9c9ecde72009b0832bdb5e" Oct 29 22:43:25.161424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:25.161392 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" event={"ID":"025ca3d2-dc28-4d52-8dc7-80869db6e538","Type":"ContainerStarted","Data":"fba26e27297cd81a98d2ab87d6daad78e0caf69c3f89f800ea724b190d110f7e"} Oct 29 22:43:25.161586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:25.161444 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" event={"ID":"025ca3d2-dc28-4d52-8dc7-80869db6e538","Type":"ContainerStarted","Data":"8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e"} Oct 29 22:43:25.216005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:25.215956 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" podStartSLOduration=2.215940712 podStartE2EDuration="2.215940712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:25.21251997 +0000 UTC m=+3666.954450642" watchObservedRunningTime="2026-10-29 22:43:25.215940712 +0000 UTC m=+3666.957871384" Oct 29 22:43:26.171291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:26.171258 4543 generic.go:356] "Generic (PLEG): container finished" podID="025ca3d2-dc28-4d52-8dc7-80869db6e538" containerID="fba26e27297cd81a98d2ab87d6daad78e0caf69c3f89f800ea724b190d110f7e" exitCode=0 Oct 29 22:43:26.171774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:26.171355 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" event={"ID":"025ca3d2-dc28-4d52-8dc7-80869db6e538","Type":"ContainerDied","Data":"fba26e27297cd81a98d2ab87d6daad78e0caf69c3f89f800ea724b190d110f7e"} Oct 29 22:43:27.360362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.360331 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:27.487984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.487950 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\") pod \"025ca3d2-dc28-4d52-8dc7-80869db6e538\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " Oct 29 22:43:27.488197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.488127 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\" (UniqueName: \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\") pod \"025ca3d2-dc28-4d52-8dc7-80869db6e538\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " Oct 29 22:43:27.488279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.488263 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\") pod \"025ca3d2-dc28-4d52-8dc7-80869db6e538\" (UID: \"025ca3d2-dc28-4d52-8dc7-80869db6e538\") " Oct 29 22:43:27.488654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.488587 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca" pod "025ca3d2-dc28-4d52-8dc7-80869db6e538" (UID: "025ca3d2-dc28-4d52-8dc7-80869db6e538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:27.489126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.489106 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.490310 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.490283 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca" pod "025ca3d2-dc28-4d52-8dc7-80869db6e538" (UID: "025ca3d2-dc28-4d52-8dc7-80869db6e538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:27.490555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.490538 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj" pod "025ca3d2-dc28-4d52-8dc7-80869db6e538" (UID: "025ca3d2-dc28-4d52-8dc7-80869db6e538"). InnerVolumeSpecName "kube-api-access-fp7tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:27.590899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.590871 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ca3d2-dc28-4d52-8dc7-80869db6e538-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.590899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:27.590898 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7tj\" (UniqueName: \"kubernetes.io/projected/025ca3d2-dc28-4d52-8dc7-80869db6e538-kube-api-access-fp7tj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.974741 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:43:27.974706 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:43:28.190412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.190379 4543 generic.go:356] "Generic (PLEG): container finished" podID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerID="9448e0f1c921c054ec3b61c5aee3ee594a78969379654cac4f895ad5f4e5af63" exitCode=0 Oct 29 22:43:28.190598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.190449 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerDied","Data":"9448e0f1c921c054ec3b61c5aee3ee594a78969379654cac4f895ad5f4e5af63"} Oct 29 22:43:28.190598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.190540 4543 scope.go:122] "RemoveContainer" containerID="177453861d86c20095c57375c07acd5cc3e6d96dbb79749d98188d5b6b241009" Oct 29 22:43:28.192280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.192252 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" event={"ID":"025ca3d2-dc28-4d52-8dc7-80869db6e538","Type":"ContainerDied","Data":"8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e"} Oct 29 22:43:28.192280 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.192279 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr" Oct 29 22:43:28.192484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.192301 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d386aca312f82cec1aee7f1bca3cf0158e5502f0bcd6e69982f5d41b6a2ba3e" Oct 29 22:43:28.452163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.452137 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:43:28.543346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543309 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543357 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543379 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543448 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543480 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543585 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543660 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543692 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:28.543776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543770 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:28.543995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543788 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.543995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.543850 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\") pod \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\" (UID: \"5ac07f82-9cbd-446e-80f6-c8a1b4202005\") " Oct 29 22:43:28.544148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544042 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:28.544254 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544233 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:28.546009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544862 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.546009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544891 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.546009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544907 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.546009 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.544930 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.546351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.546247 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:28.547694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.547662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:28.547822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.547801 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:28.547893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.547828 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "kube-api-access-zxwdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:28.547962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.547942 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo" pod "5ac07f82-9cbd-446e-80f6-c8a1b4202005" (UID: "5ac07f82-9cbd-446e-80f6-c8a1b4202005"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:28.646026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.646003 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.646026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.646025 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwdd\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-kube-api-access-zxwdd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.646218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.646034 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.646218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.646043 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:28.646218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:28.646051 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac07f82-9cbd-446e-80f6-c8a1b4202005-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:29.201409 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:29.201380 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" Oct 29 22:43:29.201570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:29.201414 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn" event={"ID":"5ac07f82-9cbd-446e-80f6-c8a1b4202005","Type":"ContainerDied","Data":"6f006a90c0b7464a157010f99a3652b0d12df7b697e96f5a10cd3a60bd2b94eb"} Oct 29 22:43:29.201570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:29.201475 4543 scope.go:122] "RemoveContainer" containerID="9448e0f1c921c054ec3b61c5aee3ee594a78969379654cac4f895ad5f4e5af63" Oct 29 22:43:29.262820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:29.262798 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:43:29.275524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:29.275499 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c06a2c-openshift-default-bf75488c7-7hsrn"] Oct 29 22:43:30.387978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:30.387945 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" path="/var/lib/kubelet/pods/5ac07f82-9cbd-446e-80f6-c8a1b4202005/volumes" Oct 29 22:43:32.075564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:32.075533 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:33.081249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:33.081206 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:42.076371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:42.076327 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:43:42.943617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:42.943591 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 22:43:43.012580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.012556 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 22:43:43.028926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.028879 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\" (UniqueName: \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\") pod \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " Oct 29 22:43:43.029064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.028945 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\" (UniqueName: \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\") pod \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " Oct 29 22:43:43.029203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.029174 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\") pod \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\" (UID: \"2a38358d-49a2-4c51-8c99-69929d3d3ef0\") " Oct 29 22:43:43.029276 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.029223 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\") pod \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\" (UID: \"9fe67fc2-a167-4c16-8af1-28d7aee4c728\") " Oct 29 22:43:43.029964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.029668 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume" pod "2a38358d-49a2-4c51-8c99-69929d3d3ef0" (UID: "2a38358d-49a2-4c51-8c99-69929d3d3ef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.029964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.029830 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume" pod "9fe67fc2-a167-4c16-8af1-28d7aee4c728" (UID: "9fe67fc2-a167-4c16-8af1-28d7aee4c728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:43.030206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.030066 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a38358d-49a2-4c51-8c99-69929d3d3ef0-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.030206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.030164 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe67fc2-a167-4c16-8af1-28d7aee4c728-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.031691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.031664 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld" pod "9fe67fc2-a167-4c16-8af1-28d7aee4c728" (UID: "9fe67fc2-a167-4c16-8af1-28d7aee4c728"). InnerVolumeSpecName "kube-api-access-frfld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:43.031909 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.031891 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8" pod "2a38358d-49a2-4c51-8c99-69929d3d3ef0" (UID: "2a38358d-49a2-4c51-8c99-69929d3d3ef0"). InnerVolumeSpecName "kube-api-access-96vz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:43.131181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.131117 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frfld\" (UniqueName: \"kubernetes.io/projected/9fe67fc2-a167-4c16-8af1-28d7aee4c728-kube-api-access-frfld\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.131181 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.131141 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vz8\" (UniqueName: \"kubernetes.io/projected/2a38358d-49a2-4c51-8c99-69929d3d3ef0-kube-api-access-96vz8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:43.312667 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.312618 4543 generic.go:356] "Generic (PLEG): container finished" podID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerID="33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd" exitCode=0 Oct 29 22:43:43.312897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.312873 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" event={"ID":"2a38358d-49a2-4c51-8c99-69929d3d3ef0","Type":"ContainerDied","Data":"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd"} Oct 29 22:43:43.313020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.312990 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" event={"ID":"2a38358d-49a2-4c51-8c99-69929d3d3ef0","Type":"ContainerDied","Data":"6410b0aa24f8fc4b49aa18473bd548ddd592dd15419457fe3298e6377f9624b9"} Oct 29 22:43:43.313130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.313037 4543 scope.go:122] "RemoveContainer" containerID="33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd" Oct 29 22:43:43.313297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.313277 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc" Oct 29 22:43:43.322238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.322044 4543 generic.go:356] "Generic (PLEG): container finished" podID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerID="07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708" exitCode=0 Oct 29 22:43:43.322238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.322154 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" event={"ID":"9fe67fc2-a167-4c16-8af1-28d7aee4c728","Type":"ContainerDied","Data":"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708"} Oct 29 22:43:43.322238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.322196 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" event={"ID":"9fe67fc2-a167-4c16-8af1-28d7aee4c728","Type":"ContainerDied","Data":"29f849e62a76df491f34cf9cd2a71e71b1f1d8037d74f199830cbb02ffebfa36"} Oct 29 22:43:43.322412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.322264 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4" Oct 29 22:43:43.333918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.333901 4543 scope.go:122] "RemoveContainer" containerID="33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd" Oct 29 22:43:43.334247 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:43:43.334221 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd\": container with ID starting with 33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd not found: ID does not exist" containerID="33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd" Oct 29 22:43:43.334323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.334259 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd"} err="failed to get container status \"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd\": rpc error: code = NotFound desc = could not find container \"33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd\": container with ID starting with 33ed086133c66545a74e6cda99797e49275b42e7093efc2001e8bb3f1a4127cd not found: ID does not exist" Oct 29 22:43:43.334323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.334280 4543 scope.go:122] "RemoveContainer" containerID="07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708" Oct 29 22:43:43.343891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.343874 4543 scope.go:122] "RemoveContainer" containerID="07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708" Oct 29 22:43:43.344296 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:43:43.344266 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708\": container with ID starting with 07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708 not found: ID does not exist" containerID="07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708" Oct 29 22:43:43.344379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.344312 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708"} err="failed to get container status \"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708\": rpc error: code = NotFound desc = could not find container \"07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708\": container with ID starting with 07b4620a8b856661ad93ba1ac3d3b44a8edb8f3eb8248b9c0f492197bea65708 not found: ID does not exist" Oct 29 22:43:43.377660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.377638 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 22:43:43.390860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.390812 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-82db63-5cbfdf9fbb-2vsm4"] Oct 29 22:43:43.429187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.429159 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 22:43:43.435491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:43.435472 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-82db63-7ddb6d6dc5-j6btc"] Oct 29 22:43:44.386165 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:44.386130 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" path="/var/lib/kubelet/pods/2a38358d-49a2-4c51-8c99-69929d3d3ef0/volumes" Oct 29 22:43:44.386564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:44.386549 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" path="/var/lib/kubelet/pods/9fe67fc2-a167-4c16-8af1-28d7aee4c728/volumes" Oct 29 22:43:47.537671 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.537633 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm"] Oct 29 22:43:47.538615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538584 4543 cpu_manager.go:414] "containerMap: removing container" podUID="025ca3d2-dc28-4d52-8dc7-80869db6e538" containerName="revoke-keys" Oct 29 22:43:47.538615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538610 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025ca3d2-dc28-4d52-8dc7-80869db6e538" containerName="revoke-keys" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538628 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538636 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538653 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538660 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538697 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538706 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538725 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" Oct 29 22:43:47.538833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538732 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" Oct 29 22:43:47.539161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538881 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.539161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538892 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a38358d-49a2-4c51-8c99-69929d3d3ef0" containerName="payload-pre-processing" Oct 29 22:43:47.539161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538905 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="025ca3d2-dc28-4d52-8dc7-80869db6e538" containerName="revoke-keys" Oct 29 22:43:47.539161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538912 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac07f82-9cbd-446e-80f6-c8a1b4202005" containerName="istio-proxy" Oct 29 22:43:47.539161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.538922 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe67fc2-a167-4c16-8af1-28d7aee4c728" containerName="payload-processing" Oct 29 22:43:47.544541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.544522 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh"] Oct 29 22:43:47.544658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.544637 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.548867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.548851 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.566679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.566652 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:43:47.575351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.575318 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:43:47.578060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578033 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.578216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578167 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.578292 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578248 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.578405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578386 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psb5\" (UniqueName: \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.578481 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578450 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fz6\" (UniqueName: \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.578572 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.578546 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.581342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.581318 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm"] Oct 29 22:43:47.587472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.587452 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh"] Oct 29 22:43:47.679986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.679959 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.680149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680026 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.680149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680121 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4psb5\" (UniqueName: \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.680227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680166 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fz6\" (UniqueName: \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.680227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.680342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.680743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.680718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.681132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.681100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.682684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.682661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.683180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.683158 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.706243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.706218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fz6\" (UniqueName: \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\") pod \"maas-api-revoke-sub-926c3146-wz4zh\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:47.706393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.706378 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psb5\" (UniqueName: \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\") pod \"maas-api-revoke-sub-ddad8b8e-bsgsm\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.863126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.863056 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:47.876980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:47.876958 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:48.217478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:48.217451 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm"] Oct 29 22:43:48.218967 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:43:48.218940 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c000f3_3fe1_4b68_8c17_c7d566919a78.slice/crio-3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc WatchSource:0}: Error finding container 3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc: Status 404 returned error can't find the container with id 3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc Oct 29 22:43:48.268848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:48.268732 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh"] Oct 29 22:43:48.371597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:48.371569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" event={"ID":"11c000f3-3fe1-4b68-8c17-c7d566919a78","Type":"ContainerStarted","Data":"3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc"} Oct 29 22:43:48.372875 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:48.372856 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" event={"ID":"f24c5eab-d7a5-442e-9938-aa32c80c48b3","Type":"ContainerStarted","Data":"4e382f89aa3cef7f66b7337ddcd3a4d39c525a1272c3ab68386a147298bd5d04"} Oct 29 22:43:49.382423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:49.382360 4543 generic.go:356] "Generic (PLEG): container finished" podID="11c000f3-3fe1-4b68-8c17-c7d566919a78" containerID="164887a4b9811964acc732e4915c10d5e865d88bac28c6784abb8e813a11aeb0" exitCode=0 Oct 29 22:43:49.382782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:49.382445 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" event={"ID":"11c000f3-3fe1-4b68-8c17-c7d566919a78","Type":"ContainerDied","Data":"164887a4b9811964acc732e4915c10d5e865d88bac28c6784abb8e813a11aeb0"} Oct 29 22:43:49.384209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:49.384188 4543 generic.go:356] "Generic (PLEG): container finished" podID="f24c5eab-d7a5-442e-9938-aa32c80c48b3" containerID="6c7adb8c50b7e7132c6d6d15fa17c649bba2b2f64e4ce68aa692b40601207f6d" exitCode=0 Oct 29 22:43:49.384318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:49.384266 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" event={"ID":"f24c5eab-d7a5-442e-9938-aa32c80c48b3","Type":"ContainerDied","Data":"6c7adb8c50b7e7132c6d6d15fa17c649bba2b2f64e4ce68aa692b40601207f6d"} Oct 29 22:43:50.580220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.580195 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:50.641100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.634977 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:50.641100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.635527 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\") pod \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " Oct 29 22:43:50.641100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.635653 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\") pod \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " Oct 29 22:43:50.641100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.635855 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\" (UniqueName: \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\") pod \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\" (UID: \"f24c5eab-d7a5-442e-9938-aa32c80c48b3\") " Oct 29 22:43:50.641100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.637848 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca" pod "f24c5eab-d7a5-442e-9938-aa32c80c48b3" (UID: "f24c5eab-d7a5-442e-9938-aa32c80c48b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:50.651287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.647298 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca" pod "f24c5eab-d7a5-442e-9938-aa32c80c48b3" (UID: "f24c5eab-d7a5-442e-9938-aa32c80c48b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:50.672954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.672919 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6" pod "f24c5eab-d7a5-442e-9938-aa32c80c48b3" (UID: "f24c5eab-d7a5-442e-9938-aa32c80c48b3"). InnerVolumeSpecName "kube-api-access-57fz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:50.737617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.737578 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\" (UniqueName: \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\") pod \"11c000f3-3fe1-4b68-8c17-c7d566919a78\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " Oct 29 22:43:50.737847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.737746 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\") pod \"11c000f3-3fe1-4b68-8c17-c7d566919a78\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " Oct 29 22:43:50.737957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.737884 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\") pod \"11c000f3-3fe1-4b68-8c17-c7d566919a78\" (UID: \"11c000f3-3fe1-4b68-8c17-c7d566919a78\") " Oct 29 22:43:50.739217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.739189 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca" pod "11c000f3-3fe1-4b68-8c17-c7d566919a78" (UID: "11c000f3-3fe1-4b68-8c17-c7d566919a78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:50.739343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.739323 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:50.739419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.739351 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24c5eab-d7a5-442e-9938-aa32c80c48b3-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:50.739419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.739367 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fz6\" (UniqueName: \"kubernetes.io/projected/f24c5eab-d7a5-442e-9938-aa32c80c48b3-kube-api-access-57fz6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:50.741303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.741279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5" pod "11c000f3-3fe1-4b68-8c17-c7d566919a78" (UID: "11c000f3-3fe1-4b68-8c17-c7d566919a78"). InnerVolumeSpecName "kube-api-access-4psb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:50.741794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.741768 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca" pod "11c000f3-3fe1-4b68-8c17-c7d566919a78" (UID: "11c000f3-3fe1-4b68-8c17-c7d566919a78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:50.840939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.840903 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4psb5\" (UniqueName: \"kubernetes.io/projected/11c000f3-3fe1-4b68-8c17-c7d566919a78-kube-api-access-4psb5\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:50.840939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.840938 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:50.841176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:50.840948 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c000f3-3fe1-4b68-8c17-c7d566919a78-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:43:51.533114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.532623 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" event={"ID":"11c000f3-3fe1-4b68-8c17-c7d566919a78","Type":"ContainerDied","Data":"3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc"} Oct 29 22:43:51.533114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.532690 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a282585386ab8624dc0a91a21f59732d7e81b22e76518f90a1b8d05d001e1fc" Oct 29 22:43:51.533114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.532765 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm" Oct 29 22:43:51.543275 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.543236 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" event={"ID":"f24c5eab-d7a5-442e-9938-aa32c80c48b3","Type":"ContainerDied","Data":"4e382f89aa3cef7f66b7337ddcd3a4d39c525a1272c3ab68386a147298bd5d04"} Oct 29 22:43:51.543412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.543307 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e382f89aa3cef7f66b7337ddcd3a4d39c525a1272c3ab68386a147298bd5d04" Oct 29 22:43:51.543412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:51.543387 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh" Oct 29 22:43:52.070168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:52.070130 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr"] Oct 29 22:43:52.070959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:52.070922 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90f46c-xgljr"] Oct 29 22:43:52.385438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:43:52.385361 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025ca3d2-dc28-4d52-8dc7-80869db6e538" path="/var/lib/kubelet/pods/025ca3d2-dc28-4d52-8dc7-80869db6e538/volumes" Oct 29 22:43:57.961916 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:43:57.961882 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:44:00.672046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672014 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj"] Oct 29 22:44:00.672690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672675 4543 cpu_manager.go:414] "containerMap: removing container" podUID="11c000f3-3fe1-4b68-8c17-c7d566919a78" containerName="revoke-keys" Oct 29 22:44:00.672737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672692 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c000f3-3fe1-4b68-8c17-c7d566919a78" containerName="revoke-keys" Oct 29 22:44:00.672737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672726 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f24c5eab-d7a5-442e-9938-aa32c80c48b3" containerName="revoke-keys" Oct 29 22:44:00.672737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672731 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24c5eab-d7a5-442e-9938-aa32c80c48b3" containerName="revoke-keys" Oct 29 22:44:00.672845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672819 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24c5eab-d7a5-442e-9938-aa32c80c48b3" containerName="revoke-keys" Oct 29 22:44:00.672845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.672832 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c000f3-3fe1-4b68-8c17-c7d566919a78" containerName="revoke-keys" Oct 29 22:44:00.676550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.676531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.679432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.679410 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn5m\" (UniqueName: \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.679521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.679458 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.679560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.679542 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.696175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.696152 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj"] Oct 29 22:44:00.705437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.705419 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:44:00.705540 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.705511 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:44:00.780907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.780883 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.781025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.780977 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.781260 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.781239 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn5m\" (UniqueName: \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.781608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.781589 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.783424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.783403 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.807459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.807432 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn5m\" (UniqueName: \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\") pod \"maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:00.990194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:00.990169 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:01.318068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:01.318037 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj"] Oct 29 22:44:01.319444 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:44:01.319419 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af3f1ca_f817_44e8_9ed8_de3f5f636b0f.slice/crio-b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6 WatchSource:0}: Error finding container b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6: Status 404 returned error can't find the container with id b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6 Oct 29 22:44:01.622831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:01.622765 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" event={"ID":"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f","Type":"ContainerStarted","Data":"b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6"} Oct 29 22:44:02.631062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:02.631025 4543 generic.go:356] "Generic (PLEG): container finished" podID="4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" containerID="cd378c03673078b7d8c0703eb2041e4ec53d0ec6a53a6602081c097a3eca2f08" exitCode=0 Oct 29 22:44:02.631455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:02.631111 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" event={"ID":"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f","Type":"ContainerDied","Data":"cd378c03673078b7d8c0703eb2041e4ec53d0ec6a53a6602081c097a3eca2f08"} Oct 29 22:44:03.808610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.808579 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:03.853062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.850962 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\") pod \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " Oct 29 22:44:03.853062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.851047 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\") pod \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " Oct 29 22:44:03.853062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.851423 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\" (UniqueName: \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\") pod \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\" (UID: \"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f\") " Oct 29 22:44:03.853567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.853536 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca" pod "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" (UID: "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:03.873097 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.858505 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca" pod "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" (UID: "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:03.908302 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.908225 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m" pod "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" (UID: "4af3f1ca-f817-44e8-9ed8-de3f5f636b0f"). InnerVolumeSpecName "kube-api-access-4tn5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:03.953318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.953277 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:03.953318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.953313 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:03.953318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:03.953323 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tn5m\" (UniqueName: \"kubernetes.io/projected/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f-kube-api-access-4tn5m\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:04.651980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:04.651890 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" Oct 29 22:44:04.651980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:04.651923 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj" event={"ID":"4af3f1ca-f817-44e8-9ed8-de3f5f636b0f","Type":"ContainerDied","Data":"b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6"} Oct 29 22:44:04.651980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:04.651975 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b067f0b7f97bc5576b9f3d4d158af97dbbf2ed8d75e3ec0e4a58118bdf1f29e6" Oct 29 22:44:11.161980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:11.161901 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:44:11.222150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:11.222123 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:44:16.163180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:16.163139 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" containerID="cri-o://5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb" gracePeriod=25 Oct 29 22:44:16.222609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:16.222577 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" containerID="cri-o://f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490" gracePeriod=25 Oct 29 22:44:23.878614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:23.878567 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b"] Oct 29 22:44:23.890626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:23.890603 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-82db63-l8l7b"] Oct 29 22:44:24.386291 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:24.386253 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ddcc861-a779-410c-9d66-89eaeeb8cb0e" path="/var/lib/kubelet/pods/8ddcc861-a779-410c-9d66-89eaeeb8cb0e/volumes" Oct 29 22:44:24.947620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:24.947585 4543 scope.go:122] "RemoveContainer" containerID="472e79b1594ed67924cf03815670103bec05da7447d2f6332134c1deae942c3d" Oct 29 22:44:24.958615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:24.958596 4543 scope.go:122] "RemoveContainer" containerID="4ec75d1f5f4ca1dfb829c614eb72ae971f5dffbe7216e74aa8576ca04557192b" Oct 29 22:44:25.243580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:25.243547 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:44:25.243731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:25.243632 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:44:27.980328 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:27.980285 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:44:31.678675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:31.678593 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 22:44:31.679160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:31.678833 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" containerID="cri-o://4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d" gracePeriod=30 Oct 29 22:44:34.263505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:34.263465 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj"] Oct 29 22:44:34.276403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:34.276378 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b7c2d5-v7hvj"] Oct 29 22:44:34.388360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:34.388327 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" path="/var/lib/kubelet/pods/4af3f1ca-f817-44e8-9ed8-de3f5f636b0f/volumes" Oct 29 22:44:35.243465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:35.243426 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:44:35.243653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:35.243424 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:44:36.496706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.496679 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:44:36.530859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.530835 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:44:36.566484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.566456 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\") pod \"b737f7de-d607-4f55-9f84-4cf609efd309\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " Oct 29 22:44:36.566630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.566611 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\" (UniqueName: \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\") pod \"b737f7de-d607-4f55-9f84-4cf609efd309\" (UID: \"b737f7de-d607-4f55-9f84-4cf609efd309\") " Oct 29 22:44:36.566776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.566763 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\") pod \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " Oct 29 22:44:36.566847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.566806 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume" pod "b737f7de-d607-4f55-9f84-4cf609efd309" (UID: "b737f7de-d607-4f55-9f84-4cf609efd309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:36.566847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.566824 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\" (UniqueName: \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\") pod \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\" (UID: \"8a95c554-1eb4-4b51-a024-79d4e1667e9c\") " Oct 29 22:44:36.567274 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.567208 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume" pod "8a95c554-1eb4-4b51-a024-79d4e1667e9c" (UID: "8a95c554-1eb4-4b51-a024-79d4e1667e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:36.567512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.567400 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a95c554-1eb4-4b51-a024-79d4e1667e9c-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.567512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.567412 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b737f7de-d607-4f55-9f84-4cf609efd309-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.568880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.568856 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7" pod "b737f7de-d607-4f55-9f84-4cf609efd309" (UID: "b737f7de-d607-4f55-9f84-4cf609efd309"). InnerVolumeSpecName "kube-api-access-xxqj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:36.569042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.568935 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb" pod "8a95c554-1eb4-4b51-a024-79d4e1667e9c" (UID: "8a95c554-1eb4-4b51-a024-79d4e1667e9c"). InnerVolumeSpecName "kube-api-access-84xxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:36.668468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.668438 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xxb\" (UniqueName: \"kubernetes.io/projected/8a95c554-1eb4-4b51-a024-79d4e1667e9c-kube-api-access-84xxb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.668685 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.668482 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqj7\" (UniqueName: \"kubernetes.io/projected/b737f7de-d607-4f55-9f84-4cf609efd309-kube-api-access-xxqj7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.925684 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.925657 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 22:44:36.964022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.963996 4543 generic.go:356] "Generic (PLEG): container finished" podID="fea783f1-8102-4110-a4d2-be7a6e509183" containerID="4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d" exitCode=0 Oct 29 22:44:36.964197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.964036 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" Oct 29 22:44:36.964197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.964088 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" event={"ID":"fea783f1-8102-4110-a4d2-be7a6e509183","Type":"ContainerDied","Data":"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d"} Oct 29 22:44:36.964197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.964140 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw" event={"ID":"fea783f1-8102-4110-a4d2-be7a6e509183","Type":"ContainerDied","Data":"fcdf6b6f53b2143917d34cd79d7109e285e0f789c2499b1416f88c549bcdd0dd"} Oct 29 22:44:36.964197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.964165 4543 scope.go:122] "RemoveContainer" containerID="4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d" Oct 29 22:44:36.965766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.965745 4543 generic.go:356] "Generic (PLEG): container finished" podID="b737f7de-d607-4f55-9f84-4cf609efd309" containerID="5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb" exitCode=0 Oct 29 22:44:36.965886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.965792 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" Oct 29 22:44:36.965886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.965813 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" event={"ID":"b737f7de-d607-4f55-9f84-4cf609efd309","Type":"ContainerDied","Data":"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb"} Oct 29 22:44:36.965886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.965879 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx" event={"ID":"b737f7de-d607-4f55-9f84-4cf609efd309","Type":"ContainerDied","Data":"12224b6b9de43acd243fe8d0fd1bde0a8289b2591a2fba15a5a6b87150f5fe9c"} Oct 29 22:44:36.967687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.967664 4543 generic.go:356] "Generic (PLEG): container finished" podID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerID="f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490" exitCode=0 Oct 29 22:44:36.967807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.967716 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" event={"ID":"8a95c554-1eb4-4b51-a024-79d4e1667e9c","Type":"ContainerDied","Data":"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490"} Oct 29 22:44:36.967807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.967754 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" event={"ID":"8a95c554-1eb4-4b51-a024-79d4e1667e9c","Type":"ContainerDied","Data":"67722ba855de31d55cf3b70c56dd7f13098980a6386dd96fc68042a266bb60da"} Oct 29 22:44:36.967904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.967823 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb" Oct 29 22:44:36.971415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971396 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971426 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971449 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971481 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971630 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971702 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:36.971817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971713 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971805 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.971998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971888 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:36.971998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\") pod \"fea783f1-8102-4110-a4d2-be7a6e509183\" (UID: \"fea783f1-8102-4110-a4d2-be7a6e509183\") " Oct 29 22:44:36.972164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.971992 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:36.972230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972187 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:36.972230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972202 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:36.972755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972728 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.972856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972761 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea783f1-8102-4110-a4d2-be7a6e509183-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.972856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972775 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.972856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972789 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.972856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.972804 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:36.974220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.974189 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:36.974220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.974213 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "kube-api-access-z48gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:36.975001 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.974978 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:36.975124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.975058 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy" pod "fea783f1-8102-4110-a4d2-be7a6e509183" (UID: "fea783f1-8102-4110-a4d2-be7a6e509183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:36.982969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.982943 4543 scope.go:122] "RemoveContainer" containerID="4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d" Oct 29 22:44:36.983362 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:36.983334 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d\": container with ID starting with 4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d not found: ID does not exist" containerID="4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d" Oct 29 22:44:36.983475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.983371 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d"} err="failed to get container status \"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d\": rpc error: code = NotFound desc = could not find container \"4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d\": container with ID starting with 4251b69faf1e2214b8370d46680d0d1f3ff43277c15c1be44ff50e68d1f58e0d not found: ID does not exist" Oct 29 22:44:36.983475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:36.983395 4543 scope.go:122] "RemoveContainer" containerID="5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb" Oct 29 22:44:37.002747 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.002714 4543 scope.go:122] "RemoveContainer" containerID="5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb" Oct 29 22:44:37.004211 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:37.004179 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb\": container with ID starting with 5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb not found: ID does not exist" containerID="5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb" Oct 29 22:44:37.004347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.004221 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb"} err="failed to get container status \"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb\": rpc error: code = NotFound desc = could not find container \"5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb\": container with ID starting with 5721b89efe8429aec9a602ab6b85a7147bd91330c4eebbcb5f76c261ac0039eb not found: ID does not exist" Oct 29 22:44:37.004347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.004244 4543 scope.go:122] "RemoveContainer" containerID="f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490" Oct 29 22:44:37.016185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.016099 4543 scope.go:122] "RemoveContainer" containerID="f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490" Oct 29 22:44:37.016424 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:37.016399 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490\": container with ID starting with f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490 not found: ID does not exist" containerID="f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490" Oct 29 22:44:37.016543 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.016433 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490"} err="failed to get container status \"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490\": rpc error: code = NotFound desc = could not find container \"f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490\": container with ID starting with f8c622b57ac5fb1aa2d6bc20ad7c4971794d169e121968c618d5914422634490 not found: ID does not exist" Oct 29 22:44:37.074570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.074545 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea783f1-8102-4110-a4d2-be7a6e509183-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.074570 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.074566 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48gj\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-kube-api-access-z48gj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.074720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.074577 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea783f1-8102-4110-a4d2-be7a6e509183-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.074720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.074585 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea783f1-8102-4110-a4d2-be7a6e509183-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.082257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.082231 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:44:37.086991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.086967 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b7c2d5-bdd7bcd67-25tdb"] Oct 29 22:44:37.126693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.126669 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:44:37.130209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.130181 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b7c2d5-55b86ff8f8-crkjx"] Oct 29 22:44:37.315176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.315148 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 22:44:37.323407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:37.323387 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-z7lbw"] Oct 29 22:44:38.386944 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:38.386911 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" path="/var/lib/kubelet/pods/8a95c554-1eb4-4b51-a024-79d4e1667e9c/volumes" Oct 29 22:44:38.387338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:38.387322 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" path="/var/lib/kubelet/pods/b737f7de-d607-4f55-9f84-4cf609efd309/volumes" Oct 29 22:44:38.387770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:38.387755 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" path="/var/lib/kubelet/pods/fea783f1-8102-4110-a4d2-be7a6e509183/volumes" Oct 29 22:44:42.239148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.239109 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:44:42.239588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.239385 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" containerID="cri-o://1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882" gracePeriod=30 Oct 29 22:44:42.822950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.822916 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc"] Oct 29 22:44:42.824586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824555 4543 cpu_manager.go:414] "containerMap: removing container" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" Oct 29 22:44:42.824811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824785 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" Oct 29 22:44:42.824926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824901 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" containerName="revoke-keys" Oct 29 22:44:42.825007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824925 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" containerName="revoke-keys" Oct 29 22:44:42.825007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824967 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" Oct 29 22:44:42.825007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.824976 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" Oct 29 22:44:42.825007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825000 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" Oct 29 22:44:42.825007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825007 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" Oct 29 22:44:42.825403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825359 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea783f1-8102-4110-a4d2-be7a6e509183" containerName="istio-proxy" Oct 29 22:44:42.825403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825403 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af3f1ca-f817-44e8-9ed8-de3f5f636b0f" containerName="revoke-keys" Oct 29 22:44:42.825547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825423 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a95c554-1eb4-4b51-a024-79d4e1667e9c" containerName="payload-pre-processing" Oct 29 22:44:42.825547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.825443 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b737f7de-d607-4f55-9f84-4cf609efd309" containerName="payload-processing" Oct 29 22:44:42.832185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.832161 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:42.844315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.844284 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc"] Oct 29 22:44:42.852179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.852143 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:44:42.852306 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.852260 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:44:42.963131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.963106 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:42.963320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.963297 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:42.963390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:42.963355 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jn6\" (UniqueName: \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.064216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.064193 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jn6\" (UniqueName: \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.064343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.064255 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.064453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.064434 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.065027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.065010 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.066784 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.066764 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.091803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.091755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jn6\" (UniqueName: \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.154712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.154691 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:43.462227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:43.462198 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc"] Oct 29 22:44:43.463888 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:44:43.463860 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a713914_26c2_4d10_8a45_dbd0b2b2ce17.slice/crio-1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa WatchSource:0}: Error finding container 1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa: Status 404 returned error can't find the container with id 1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa Oct 29 22:44:44.029043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:44.028859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" event={"ID":"4a713914-26c2-4d10-8a45-dbd0b2b2ce17","Type":"ContainerStarted","Data":"2bff7e23b3d75dcd0672b90cee3e20858e3bc6cb486df2f7f26f98a7fe0db278"} Oct 29 22:44:44.029043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:44.028927 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" event={"ID":"4a713914-26c2-4d10-8a45-dbd0b2b2ce17","Type":"ContainerStarted","Data":"1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa"} Oct 29 22:44:44.075991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:44.075946 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" podStartSLOduration=2.07593335 podStartE2EDuration="2.07593335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:44.074175675 +0000 UTC m=+3745.816106347" watchObservedRunningTime="2026-10-29 22:44:44.07593335 +0000 UTC m=+3745.817864023" Oct 29 22:44:45.038379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:45.038345 4543 generic.go:356] "Generic (PLEG): container finished" podID="4a713914-26c2-4d10-8a45-dbd0b2b2ce17" containerID="2bff7e23b3d75dcd0672b90cee3e20858e3bc6cb486df2f7f26f98a7fe0db278" exitCode=0 Oct 29 22:44:45.038757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:45.038430 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" event={"ID":"4a713914-26c2-4d10-8a45-dbd0b2b2ce17","Type":"ContainerDied","Data":"2bff7e23b3d75dcd0672b90cee3e20858e3bc6cb486df2f7f26f98a7fe0db278"} Oct 29 22:44:46.096863 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.096825 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:44:46.097293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.096928 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:44:46.191836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.191812 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:46.300502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.300439 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\" (UniqueName: \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\") pod \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " Oct 29 22:44:46.300640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.300544 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\") pod \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " Oct 29 22:44:46.300640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.300607 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\") pod \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\" (UID: \"4a713914-26c2-4d10-8a45-dbd0b2b2ce17\") " Oct 29 22:44:46.301044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.300999 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca" pod "4a713914-26c2-4d10-8a45-dbd0b2b2ce17" (UID: "4a713914-26c2-4d10-8a45-dbd0b2b2ce17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:46.302691 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.302663 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca" pod "4a713914-26c2-4d10-8a45-dbd0b2b2ce17" (UID: "4a713914-26c2-4d10-8a45-dbd0b2b2ce17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:46.302787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.302756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6" pod "4a713914-26c2-4d10-8a45-dbd0b2b2ce17" (UID: "4a713914-26c2-4d10-8a45-dbd0b2b2ce17"). InnerVolumeSpecName "kube-api-access-t2jn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:46.402349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.402322 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jn6\" (UniqueName: \"kubernetes.io/projected/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-kube-api-access-t2jn6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:46.402349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.402347 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:46.402492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:46.402357 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a713914-26c2-4d10-8a45-dbd0b2b2ce17-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.060011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.059978 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" Oct 29 22:44:47.060297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.060006 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc" event={"ID":"4a713914-26c2-4d10-8a45-dbd0b2b2ce17","Type":"ContainerDied","Data":"1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa"} Oct 29 22:44:47.060297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.060054 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8512f52af258580603f8fc9d0357f1953ebcd7e54414b14640887ffac44dfa" Oct 29 22:44:47.531215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.531192 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:44:47.620482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620407 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620479 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620511 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620610 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620696 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620779 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.620888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620813 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:47.621158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620916 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.621158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.620967 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\") pod \"9560df2f-92d0-474d-b279-283cb493c35c\" (UID: \"9560df2f-92d0-474d-b279-283cb493c35c\") " Oct 29 22:44:47.621158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621067 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:47.621315 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621259 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:47.621499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:47.621705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621684 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.622035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621712 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.622035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621728 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9560df2f-92d0-474d-b279-283cb493c35c-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.622035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621754 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.622035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.621710 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:47.624645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.624606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:47.624645 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.624617 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "kube-api-access-kz4jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:47.624771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.624666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:47.626278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.626253 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy" pod "9560df2f-92d0-474d-b279-283cb493c35c" (UID: "9560df2f-92d0-474d-b279-283cb493c35c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:47.723286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.723253 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.723286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.723281 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4jn\" (UniqueName: \"kubernetes.io/projected/9560df2f-92d0-474d-b279-283cb493c35c-kube-api-access-kz4jn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.723286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.723292 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.723507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.723312 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9560df2f-92d0-474d-b279-283cb493c35c-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:47.723507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:47.723324 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9560df2f-92d0-474d-b279-283cb493c35c-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:48.069094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.069043 4543 generic.go:356] "Generic (PLEG): container finished" podID="9560df2f-92d0-474d-b279-283cb493c35c" containerID="1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882" exitCode=0 Oct 29 22:44:48.069258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.069123 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" Oct 29 22:44:48.069258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.069126 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerDied","Data":"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882"} Oct 29 22:44:48.069258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.069176 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn" event={"ID":"9560df2f-92d0-474d-b279-283cb493c35c","Type":"ContainerDied","Data":"0dd9aa2b382afcc0adc3ec0f4ea32813a91a967afa21049eb87618b9b98efc50"} Oct 29 22:44:48.069258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.069200 4543 scope.go:122] "RemoveContainer" containerID="1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882" Oct 29 22:44:48.081020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.080999 4543 scope.go:122] "RemoveContainer" containerID="8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0" Oct 29 22:44:48.091728 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.091712 4543 scope.go:122] "RemoveContainer" containerID="1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882" Oct 29 22:44:48.091993 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:48.091973 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882\": container with ID starting with 1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882 not found: ID does not exist" containerID="1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882" Oct 29 22:44:48.092102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.092003 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882"} err="failed to get container status \"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882\": rpc error: code = NotFound desc = could not find container \"1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882\": container with ID starting with 1400fd8aa7a6dceb60825a6e5e8f70d67f52ab2abb498ea2e57e804efa7b0882 not found: ID does not exist" Oct 29 22:44:48.092102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.092028 4543 scope.go:122] "RemoveContainer" containerID="8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0" Oct 29 22:44:48.092273 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:48.092256 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0\": container with ID starting with 8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0 not found: ID does not exist" containerID="8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0" Oct 29 22:44:48.092323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.092280 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0"} err="failed to get container status \"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0\": rpc error: code = NotFound desc = could not find container \"8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0\": container with ID starting with 8457f7ee187527c88be384c9bc1148e90a59c18beda4c9271b6cc050f7b696b0 not found: ID does not exist" Oct 29 22:44:48.129633 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.129608 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:44:48.142925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.142893 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b7c2d5-openshift-default-f76695d8f-kgrcn"] Oct 29 22:44:48.395441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:48.395369 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9560df2f-92d0-474d-b279-283cb493c35c" path="/var/lib/kubelet/pods/9560df2f-92d0-474d-b279-283cb493c35c/volumes" Oct 29 22:44:52.349565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.349533 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4"] Oct 29 22:44:52.350223 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350207 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350224 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350247 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350252 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350270 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4a713914-26c2-4d10-8a45-dbd0b2b2ce17" containerName="revoke-keys" Oct 29 22:44:52.350278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350275 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a713914-26c2-4d10-8a45-dbd0b2b2ce17" containerName="revoke-keys" Oct 29 22:44:52.350446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350370 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a713914-26c2-4d10-8a45-dbd0b2b2ce17" containerName="revoke-keys" Oct 29 22:44:52.350446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350385 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.350446 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.350394 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9560df2f-92d0-474d-b279-283cb493c35c" containerName="istio-proxy" Oct 29 22:44:52.355349 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.355334 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.375139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.375115 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4"] Oct 29 22:44:52.383718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.383698 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:44:52.383827 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.383782 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:44:52.473907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.473878 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5xp\" (UniqueName: \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.474025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.473926 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.474070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.474023 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.575096 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.575051 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5xp\" (UniqueName: \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.575203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.575117 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.575203 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.575165 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.575807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.575784 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.577637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.577615 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.602169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.602111 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5xp\" (UniqueName: \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\") pod \"maas-api-revoke-sub-effe9389-4mjc4\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.669472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.669451 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:52.977159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:52.977131 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4"] Oct 29 22:44:52.978724 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:44:52.978691 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb29b465_7dcc_424a_a70a_a684b260b775.slice/crio-9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06 WatchSource:0}: Error finding container 9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06: Status 404 returned error can't find the container with id 9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06 Oct 29 22:44:53.109406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:53.109379 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" event={"ID":"db29b465-7dcc-424a-a70a-a684b260b775","Type":"ContainerStarted","Data":"9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06"} Oct 29 22:44:53.305664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:53.305637 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:44:53.361717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:53.361693 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:44:54.122493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:54.122400 4543 generic.go:356] "Generic (PLEG): container finished" podID="db29b465-7dcc-424a-a70a-a684b260b775" containerID="0216c05c9167f16faa7a501b925447b06c4a1563bdb932d43ad26a6cd906876e" exitCode=0 Oct 29 22:44:54.122683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:54.122481 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" event={"ID":"db29b465-7dcc-424a-a70a-a684b260b775","Type":"ContainerDied","Data":"0216c05c9167f16faa7a501b925447b06c4a1563bdb932d43ad26a6cd906876e"} Oct 29 22:44:55.275640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.275615 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:55.425589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.425400 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\") pod \"db29b465-7dcc-424a-a70a-a684b260b775\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " Oct 29 22:44:55.425951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.425927 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\") pod \"db29b465-7dcc-424a-a70a-a684b260b775\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " Oct 29 22:44:55.426136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.426116 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\" (UniqueName: \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\") pod \"db29b465-7dcc-424a-a70a-a684b260b775\" (UID: \"db29b465-7dcc-424a-a70a-a684b260b775\") " Oct 29 22:44:55.432873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.432842 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca" pod "db29b465-7dcc-424a-a70a-a684b260b775" (UID: "db29b465-7dcc-424a-a70a-a684b260b775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:55.442022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.441976 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca" pod "db29b465-7dcc-424a-a70a-a684b260b775" (UID: "db29b465-7dcc-424a-a70a-a684b260b775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:55.442317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.442293 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp" pod "db29b465-7dcc-424a-a70a-a684b260b775" (UID: "db29b465-7dcc-424a-a70a-a684b260b775"). InnerVolumeSpecName "kube-api-access-7l5xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:55.527731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.527704 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.527731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.527729 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db29b465-7dcc-424a-a70a-a684b260b775-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:55.527912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:55.527739 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5xp\" (UniqueName: \"kubernetes.io/projected/db29b465-7dcc-424a-a70a-a684b260b775-kube-api-access-7l5xp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:44:56.139731 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:56.139704 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" Oct 29 22:44:56.139898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:56.139737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4" event={"ID":"db29b465-7dcc-424a-a70a-a684b260b775","Type":"ContainerDied","Data":"9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06"} Oct 29 22:44:56.139898 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:56.139783 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fba9e3eea9d5faf6298e76b99d6a7a0ea1239f5318197bd0f91fb23f4a4bd06" Oct 29 22:44:57.960675 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:44:57.960635 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:44:58.306360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:58.306309 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" containerID="cri-o://fb7bc2b854462de810345e048b4588ac5d84a6cb23ddd64d822ec778e13e4779" gracePeriod=25 Oct 29 22:44:58.362896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:44:58.362853 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" containerID="cri-o://09b517d0c4ca8a301faebc3505b4f8e764db07ff63ed91d30febe49cde806a87" gracePeriod=25 Oct 29 22:45:00.155853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.155826 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:00.156518 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.156501 4543 cpu_manager.go:414] "containerMap: removing container" podUID="db29b465-7dcc-424a-a70a-a684b260b775" containerName="revoke-keys" Oct 29 22:45:00.156565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.156520 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db29b465-7dcc-424a-a70a-a684b260b775" containerName="revoke-keys" Oct 29 22:45:00.156622 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.156613 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="db29b465-7dcc-424a-a70a-a684b260b775" containerName="revoke-keys" Oct 29 22:45:00.160657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.160635 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:00.160802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.160755 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.164879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.164859 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:00.165018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.164986 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.170383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.170367 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.186498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.186477 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:00.191782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.191761 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:00.192290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.192268 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.194739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.194719 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:00.199252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.198724 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:00.199252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.198813 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.203676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.203655 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:00.272842 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.272816 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:00.279267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.279424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279267 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldvc\" (UniqueName: \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.279424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279305 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jr8z\" (UniqueName: \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.279424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279385 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.279600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.279600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.279560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdf7\" (UniqueName: \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.288988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.288962 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:00.296284 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.296261 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:00.296537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.296362 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.306862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.306846 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:00.306998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.306988 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:00.307155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.307145 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.308066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.308048 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:00.311290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.311268 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:00.316581 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.316563 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:00.316675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.316659 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.319228 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.319209 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:00.380661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.380634 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.380789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.380666 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldvc\" (UniqueName: \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.380789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.380684 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jr8z\" (UniqueName: \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.380876 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.380817 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswz5\" (UniqueName: \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.380929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.380870 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.381031 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381014 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.381122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9s7t\" (UniqueName: \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.381122 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381108 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.381230 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.381269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381244 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdf7\" (UniqueName: \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.381352 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381334 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.381741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381719 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.381867 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.381848 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.407033 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.406968 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdf7\" (UniqueName: \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\") pod \"maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.407190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.407026 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldvc\" (UniqueName: \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\") pod \"maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.407268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.407217 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jr8z\" (UniqueName: \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\") pod \"maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.476866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.476843 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:00.482265 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482244 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6wp\" (UniqueName: \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.482371 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482286 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.482460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482445 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.482523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482489 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswz5\" (UniqueName: \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.482624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482604 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.482697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482665 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6gp\" (UniqueName: \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.482752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482699 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.482877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482855 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9s7t\" (UniqueName: \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.482922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482896 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.483012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.482996 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv84w\" (UniqueName: \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.483328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.483312 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.483527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.483511 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.488087 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.488054 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:00.499393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.499246 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:00.508787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.508754 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswz5\" (UniqueName: \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.509173 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.509040 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9s7t\" (UniqueName: \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.559681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.559187 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.575805 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.584751 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6gp\" (UniqueName: \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.584794 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.584877 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv84w\" (UniqueName: \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.584920 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6wp\" (UniqueName: \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.584943 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.585493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.585021 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.586369 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.585778 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.587226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.586858 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.588717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.587749 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.614299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.614252 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv84w\" (UniqueName: \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.615686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.615640 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6gp\" (UniqueName: \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.615686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.615647 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6wp\" (UniqueName: \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.637250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.636950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:00.649117 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.648880 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:00.806624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.806540 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:00.807392 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:00.807352 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b34944_ce0b_4b0f_aee5_06d2b3559817.slice/crio-388176f700f68f19d77be8b287d2392e50e96a849451eb5a5686a41d0f094a9f WatchSource:0}: Error finding container 388176f700f68f19d77be8b287d2392e50e96a849451eb5a5686a41d0f094a9f: Status 404 returned error can't find the container with id 388176f700f68f19d77be8b287d2392e50e96a849451eb5a5686a41d0f094a9f Oct 29 22:45:00.841828 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.841801 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:00.845315 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:00.845288 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d5daf2_a81e_4d8c_beb1_3ced71afc77c.slice/crio-2a7805724491d2ab64a25c7f4bd3d1a14998cbccd0340df365477fff2e5ce03a WatchSource:0}: Error finding container 2a7805724491d2ab64a25c7f4bd3d1a14998cbccd0340df365477fff2e5ce03a: Status 404 returned error can't find the container with id 2a7805724491d2ab64a25c7f4bd3d1a14998cbccd0340df365477fff2e5ce03a Oct 29 22:45:00.914465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.914410 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:00.946403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:00.945751 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:01.029485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.029220 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:01.032270 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.032237 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98356f37_0e49_4513_99a2_d2f7c7f85ca7.slice/crio-9fc145b206abff59e2e6f9274d53b79d9446b5b57672838f18ba8b6de3dae722 WatchSource:0}: Error finding container 9fc145b206abff59e2e6f9274d53b79d9446b5b57672838f18ba8b6de3dae722: Status 404 returned error can't find the container with id 9fc145b206abff59e2e6f9274d53b79d9446b5b57672838f18ba8b6de3dae722 Oct 29 22:45:01.033937 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.033912 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084b405f_b470_4fb4_99e8_a99bddd9bc56.slice/crio-2c0c2e0b600d1f17d5ca8b1089a412f2127dd9242b984aec3a9560defa35aefb WatchSource:0}: Error finding container 2c0c2e0b600d1f17d5ca8b1089a412f2127dd9242b984aec3a9560defa35aefb: Status 404 returned error can't find the container with id 2c0c2e0b600d1f17d5ca8b1089a412f2127dd9242b984aec3a9560defa35aefb Oct 29 22:45:01.035387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.035359 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:01.044221 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.044191 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f2cd4c_eb99_4326_944d_a24aa7385d8f.slice/crio-836fc0c862e224e87dc26b9b99c798655d28845dc11d78d0370a0b5d58b5c013 WatchSource:0}: Error finding container 836fc0c862e224e87dc26b9b99c798655d28845dc11d78d0370a0b5d58b5c013: Status 404 returned error can't find the container with id 836fc0c862e224e87dc26b9b99c798655d28845dc11d78d0370a0b5d58b5c013 Oct 29 22:45:01.057340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.057316 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:01.062561 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.061186 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcc7020_9571_4469_bc7c_1f496ac7a0f6.slice/crio-dd2265b3e23a132f0c32a352663b95a34a8af65b7056e339d951d11a77258098 WatchSource:0}: Error finding container dd2265b3e23a132f0c32a352663b95a34a8af65b7056e339d951d11a77258098: Status 404 returned error can't find the container with id dd2265b3e23a132f0c32a352663b95a34a8af65b7056e339d951d11a77258098 Oct 29 22:45:01.117894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.117866 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:01.119119 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.119069 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ceda9a6_192b_4e99_ba66_2528b3161976.slice/crio-209a3c6f74636b72d6d55ed36d7189db8dcf47c26c4d479d240d0915a93f9ba7 WatchSource:0}: Error finding container 209a3c6f74636b72d6d55ed36d7189db8dcf47c26c4d479d240d0915a93f9ba7: Status 404 returned error can't find the container with id 209a3c6f74636b72d6d55ed36d7189db8dcf47c26c4d479d240d0915a93f9ba7 Oct 29 22:45:01.186126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.186051 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerStarted","Data":"209a3c6f74636b72d6d55ed36d7189db8dcf47c26c4d479d240d0915a93f9ba7"} Oct 29 22:45:01.187538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.187506 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerStarted","Data":"836fc0c862e224e87dc26b9b99c798655d28845dc11d78d0370a0b5d58b5c013"} Oct 29 22:45:01.188800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.188778 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerStarted","Data":"2a7805724491d2ab64a25c7f4bd3d1a14998cbccd0340df365477fff2e5ce03a"} Oct 29 22:45:01.190121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.190098 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerStarted","Data":"9fc145b206abff59e2e6f9274d53b79d9446b5b57672838f18ba8b6de3dae722"} Oct 29 22:45:01.191475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.191451 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerStarted","Data":"2c0c2e0b600d1f17d5ca8b1089a412f2127dd9242b984aec3a9560defa35aefb"} Oct 29 22:45:01.192676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.192656 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerStarted","Data":"dd2265b3e23a132f0c32a352663b95a34a8af65b7056e339d951d11a77258098"} Oct 29 22:45:01.193946 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.193926 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerStarted","Data":"388176f700f68f19d77be8b287d2392e50e96a849451eb5a5686a41d0f094a9f"} Oct 29 22:45:01.333953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:01.333876 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:01.432446 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:01.432159 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac5cbce_f0b2_4642_980f_c918af7c0747.slice/crio-3d38eaca6ea635266f0a17190ddee4699c141e82e8c99fee0aba32044f5f1bea WatchSource:0}: Error finding container 3d38eaca6ea635266f0a17190ddee4699c141e82e8c99fee0aba32044f5f1bea: Status 404 returned error can't find the container with id 3d38eaca6ea635266f0a17190ddee4699c141e82e8c99fee0aba32044f5f1bea Oct 29 22:45:02.123273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.123238 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:02.123438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.123399 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:02.204359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.204280 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerStarted","Data":"98cd0cefc9124c393b099cec48841cde07c984457a254744d4e14c156077d443"} Oct 29 22:45:02.206022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.205989 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerStarted","Data":"fe03cf012bf636e5d6d6d9160a90c6e96e6159967afe5945d85f2424eedcc94b"} Oct 29 22:45:02.207509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.207484 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerStarted","Data":"03413d9233eb7b28d93b3c0f6d3a8f43dc650b6201d27c76f602f516e9c6fb85"} Oct 29 22:45:02.207613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.207532 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerStarted","Data":"3d38eaca6ea635266f0a17190ddee4699c141e82e8c99fee0aba32044f5f1bea"} Oct 29 22:45:02.209032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.209010 4543 generic.go:356] "Generic (PLEG): container finished" podID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerID="3a81ff528d9d01d8bb974f349510ed0390daa92a0c82ce42c5418cb7a040cb6c" exitCode=60 Oct 29 22:45:02.209144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.209086 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerDied","Data":"3a81ff528d9d01d8bb974f349510ed0390daa92a0c82ce42c5418cb7a040cb6c"} Oct 29 22:45:02.209296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.209282 4543 scope.go:122] "RemoveContainer" containerID="3a81ff528d9d01d8bb974f349510ed0390daa92a0c82ce42c5418cb7a040cb6c" Oct 29 22:45:02.210688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.210658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerStarted","Data":"86575e0e9f707b33db6d0b71ab87bf1e544be8bbe2d5a43928559e0447c4ed3b"} Oct 29 22:45:02.212395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.212373 4543 generic.go:356] "Generic (PLEG): container finished" podID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerID="a9177ab520db572a4fdd45a5edfbb01f5792560197d4e22c552c61f1ff3b35be" exitCode=60 Oct 29 22:45:02.212483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.212434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerDied","Data":"a9177ab520db572a4fdd45a5edfbb01f5792560197d4e22c552c61f1ff3b35be"} Oct 29 22:45:02.212668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.212616 4543 scope.go:122] "RemoveContainer" containerID="a9177ab520db572a4fdd45a5edfbb01f5792560197d4e22c552c61f1ff3b35be" Oct 29 22:45:02.214383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.214358 4543 generic.go:356] "Generic (PLEG): container finished" podID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerID="bfb66e45652fcba30fbfccb3e66f78a9a75b5027f02f2b17487b726b743b469c" exitCode=60 Oct 29 22:45:02.214456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.214434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerDied","Data":"bfb66e45652fcba30fbfccb3e66f78a9a75b5027f02f2b17487b726b743b469c"} Oct 29 22:45:02.214603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.214585 4543 scope.go:122] "RemoveContainer" containerID="bfb66e45652fcba30fbfccb3e66f78a9a75b5027f02f2b17487b726b743b469c" Oct 29 22:45:02.216216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.216194 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerStarted","Data":"4b06eefdf804906a3837b870e85a4f81dd1591a71031beef2bc34fe50d126aa3"} Oct 29 22:45:02.250652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.250608 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" podStartSLOduration=2.250594735 podStartE2EDuration="2.250594735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.247604748 +0000 UTC m=+3763.989535412" watchObservedRunningTime="2026-10-29 22:45:02.250594735 +0000 UTC m=+3763.992525407" Oct 29 22:45:02.385061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.385001 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" podStartSLOduration=2.384981815 podStartE2EDuration="2.384981815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.38346947 +0000 UTC m=+3764.125400144" watchObservedRunningTime="2026-10-29 22:45:02.384981815 +0000 UTC m=+3764.126912489" Oct 29 22:45:02.426894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.426810 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" podStartSLOduration=2.426798128 podStartE2EDuration="2.426798128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.426691799 +0000 UTC m=+3764.168622472" watchObservedRunningTime="2026-10-29 22:45:02.426798128 +0000 UTC m=+3764.168728801" Oct 29 22:45:02.472615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.472571 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" podStartSLOduration=2.472557228 podStartE2EDuration="2.472557228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.469331064 +0000 UTC m=+3764.211261734" watchObservedRunningTime="2026-10-29 22:45:02.472557228 +0000 UTC m=+3764.214487900" Oct 29 22:45:02.562505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:02.562463 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" podStartSLOduration=2.56245086 podStartE2EDuration="2.56245086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.561188659 +0000 UTC m=+3764.303119332" watchObservedRunningTime="2026-10-29 22:45:02.56245086 +0000 UTC m=+3764.304381532" Oct 29 22:45:03.225725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.225693 4543 generic.go:356] "Generic (PLEG): container finished" podID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerID="98cd0cefc9124c393b099cec48841cde07c984457a254744d4e14c156077d443" exitCode=60 Oct 29 22:45:03.226144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.225772 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerDied","Data":"98cd0cefc9124c393b099cec48841cde07c984457a254744d4e14c156077d443"} Oct 29 22:45:03.226144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.226065 4543 scope.go:122] "RemoveContainer" containerID="98cd0cefc9124c393b099cec48841cde07c984457a254744d4e14c156077d443" Oct 29 22:45:03.227569 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.227542 4543 generic.go:356] "Generic (PLEG): container finished" podID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerID="fe03cf012bf636e5d6d6d9160a90c6e96e6159967afe5945d85f2424eedcc94b" exitCode=60 Oct 29 22:45:03.227682 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.227624 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerDied","Data":"fe03cf012bf636e5d6d6d9160a90c6e96e6159967afe5945d85f2424eedcc94b"} Oct 29 22:45:03.227912 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.227897 4543 scope.go:122] "RemoveContainer" containerID="fe03cf012bf636e5d6d6d9160a90c6e96e6159967afe5945d85f2424eedcc94b" Oct 29 22:45:03.230143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.230115 4543 generic.go:356] "Generic (PLEG): container finished" podID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerID="03413d9233eb7b28d93b3c0f6d3a8f43dc650b6201d27c76f602f516e9c6fb85" exitCode=60 Oct 29 22:45:03.230256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.230238 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerDied","Data":"03413d9233eb7b28d93b3c0f6d3a8f43dc650b6201d27c76f602f516e9c6fb85"} Oct 29 22:45:03.230744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.230721 4543 scope.go:122] "RemoveContainer" containerID="03413d9233eb7b28d93b3c0f6d3a8f43dc650b6201d27c76f602f516e9c6fb85" Oct 29 22:45:03.232573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.232550 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerStarted","Data":"07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19"} Oct 29 22:45:03.234494 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.234473 4543 generic.go:356] "Generic (PLEG): container finished" podID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerID="86575e0e9f707b33db6d0b71ab87bf1e544be8bbe2d5a43928559e0447c4ed3b" exitCode=60 Oct 29 22:45:03.234589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.234515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerDied","Data":"86575e0e9f707b33db6d0b71ab87bf1e544be8bbe2d5a43928559e0447c4ed3b"} Oct 29 22:45:03.234753 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.234734 4543 scope.go:122] "RemoveContainer" containerID="86575e0e9f707b33db6d0b71ab87bf1e544be8bbe2d5a43928559e0447c4ed3b" Oct 29 22:45:03.237550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.237123 4543 generic.go:356] "Generic (PLEG): container finished" podID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerID="26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4" exitCode=60 Oct 29 22:45:03.237550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.237210 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerDied","Data":"26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4"} Oct 29 22:45:03.237550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.237263 4543 scope.go:122] "RemoveContainer" containerID="a9177ab520db572a4fdd45a5edfbb01f5792560197d4e22c552c61f1ff3b35be" Oct 29 22:45:03.237550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.237525 4543 scope.go:122] "RemoveContainer" containerID="26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4" Oct 29 22:45:03.237750 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:03.237727 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8_odh-ai-gateway-infra(39f2cd4c-eb99-4326-944d-a24aa7385d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" Oct 29 22:45:03.240437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.240411 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerStarted","Data":"5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e"} Oct 29 22:45:03.242287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.242268 4543 generic.go:356] "Generic (PLEG): container finished" podID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerID="4b06eefdf804906a3837b870e85a4f81dd1591a71031beef2bc34fe50d126aa3" exitCode=60 Oct 29 22:45:03.242375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.242315 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerDied","Data":"4b06eefdf804906a3837b870e85a4f81dd1591a71031beef2bc34fe50d126aa3"} Oct 29 22:45:03.242505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.242492 4543 scope.go:122] "RemoveContainer" containerID="4b06eefdf804906a3837b870e85a4f81dd1591a71031beef2bc34fe50d126aa3" Oct 29 22:45:03.403647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.403575 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" podStartSLOduration=3.403560289 podStartE2EDuration="3.403560289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.403232955 +0000 UTC m=+3765.145163640" watchObservedRunningTime="2026-10-29 22:45:03.403560289 +0000 UTC m=+3765.145490962" Oct 29 22:45:03.547980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.547933 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" podStartSLOduration=3.547918402 podStartE2EDuration="3.547918402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.54452526 +0000 UTC m=+3765.286455945" watchObservedRunningTime="2026-10-29 22:45:03.547918402 +0000 UTC m=+3765.289849075" Oct 29 22:45:03.928110 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.928059 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s"] Oct 29 22:45:03.933871 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.933851 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:03.953844 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.953818 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s"] Oct 29 22:45:03.956168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.956148 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:03.975070 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.975046 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:03.975272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.975249 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:03.975338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:03.975295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkdx\" (UniqueName: \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.076060 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.075996 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.076208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.076114 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.076208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.076137 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkdx\" (UniqueName: \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.076686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.076660 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.078555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.078528 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.101577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.101556 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkdx\" (UniqueName: \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\") pod \"maas-api-revoke-sub-3522aa13-c4x5s\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.248801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.248761 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:04.283553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.283523 4543 scope.go:122] "RemoveContainer" containerID="26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4" Oct 29 22:45:04.283915 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:04.283888 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8_odh-ai-gateway-infra(39f2cd4c-eb99-4326-944d-a24aa7385d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" Oct 29 22:45:04.285800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.285773 4543 generic.go:356] "Generic (PLEG): container finished" podID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerID="5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e" exitCode=60 Oct 29 22:45:04.285943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.285865 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerDied","Data":"5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e"} Oct 29 22:45:04.285943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.285927 4543 scope.go:122] "RemoveContainer" containerID="bfb66e45652fcba30fbfccb3e66f78a9a75b5027f02f2b17487b726b743b469c" Oct 29 22:45:04.286172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.286154 4543 scope.go:122] "RemoveContainer" containerID="5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e" Oct 29 22:45:04.286437 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:04.286418 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr_odh-ai-gateway-infra(d3d5daf2-a81e-4d8c-beb1-3ced71afc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" Oct 29 22:45:04.288472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.288446 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerStarted","Data":"8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8"} Oct 29 22:45:04.288760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.288744 4543 scope.go:122] "RemoveContainer" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" Oct 29 22:45:04.289062 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:04.289025 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b_odh-ai-gateway-infra(98356f37-0e49-4513-99a2-d2f7c7f85ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" Oct 29 22:45:04.292035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.292007 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerStarted","Data":"f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70"} Oct 29 22:45:04.294951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.294930 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerStarted","Data":"ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a"} Oct 29 22:45:04.295168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.295154 4543 scope.go:122] "RemoveContainer" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" Oct 29 22:45:04.295394 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:04.295375 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm_odh-ai-gateway-infra(ffcc7020-9571-4469-bc7c-1f496ac7a0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" Oct 29 22:45:04.297688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.297662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerStarted","Data":"acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa"} Oct 29 22:45:04.300279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.300255 4543 generic.go:356] "Generic (PLEG): container finished" podID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerID="07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19" exitCode=60 Oct 29 22:45:04.300381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.300369 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerDied","Data":"07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19"} Oct 29 22:45:04.300636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.300617 4543 scope.go:122] "RemoveContainer" containerID="07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19" Oct 29 22:45:04.300869 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:04.300850 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf_odh-ai-gateway-infra(73b34944-ce0b-4b0f-aee5-06d2b3559817)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" Oct 29 22:45:04.302930 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.302907 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerStarted","Data":"f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2"} Oct 29 22:45:04.440222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.438852 4543 scope.go:122] "RemoveContainer" containerID="3a81ff528d9d01d8bb974f349510ed0390daa92a0c82ce42c5418cb7a040cb6c" Oct 29 22:45:04.597657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:04.597597 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s"] Oct 29 22:45:04.602853 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:04.602824 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8630ee4_7d00_4f36_95a4_634e3127c263.slice/crio-782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e WatchSource:0}: Error finding container 782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e: Status 404 returned error can't find the container with id 782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e Oct 29 22:45:05.313143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.313109 4543 scope.go:122] "RemoveContainer" containerID="5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e" Oct 29 22:45:05.313574 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.313400 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr_odh-ai-gateway-infra(d3d5daf2-a81e-4d8c-beb1-3ced71afc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" Oct 29 22:45:05.314720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.314694 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" event={"ID":"d8630ee4-7d00-4f36-95a4-634e3127c263","Type":"ContainerStarted","Data":"213ef68f1b3617114ed30b706de8a86ccb972be98e0124bb2281ba4dec1ea1a7"} Oct 29 22:45:05.314851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.314751 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" event={"ID":"d8630ee4-7d00-4f36-95a4-634e3127c263","Type":"ContainerStarted","Data":"782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e"} Oct 29 22:45:05.316758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.316732 4543 generic.go:356] "Generic (PLEG): container finished" podID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" exitCode=60 Oct 29 22:45:05.316869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.316772 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerDied","Data":"8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8"} Oct 29 22:45:05.316869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.316832 4543 scope.go:122] "RemoveContainer" containerID="4b06eefdf804906a3837b870e85a4f81dd1591a71031beef2bc34fe50d126aa3" Oct 29 22:45:05.317049 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.317026 4543 scope.go:122] "RemoveContainer" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" Oct 29 22:45:05.317313 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.317287 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b_odh-ai-gateway-infra(98356f37-0e49-4513-99a2-d2f7c7f85ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" Oct 29 22:45:05.318911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.318887 4543 generic.go:356] "Generic (PLEG): container finished" podID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerID="f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70" exitCode=60 Oct 29 22:45:05.319003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.318959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerDied","Data":"f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70"} Oct 29 22:45:05.319241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.319223 4543 scope.go:122] "RemoveContainer" containerID="f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70" Oct 29 22:45:05.319458 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.319435 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl_odh-ai-gateway-infra(084b405f-b470-4fb4-99e8-a99bddd9bc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" Oct 29 22:45:05.320869 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.320847 4543 generic.go:356] "Generic (PLEG): container finished" podID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" exitCode=60 Oct 29 22:45:05.320971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.320951 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerDied","Data":"ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a"} Oct 29 22:45:05.321171 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.321152 4543 scope.go:122] "RemoveContainer" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" Oct 29 22:45:05.321408 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.321390 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm_odh-ai-gateway-infra(ffcc7020-9571-4469-bc7c-1f496ac7a0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" Oct 29 22:45:05.323099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.323057 4543 generic.go:356] "Generic (PLEG): container finished" podID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerID="acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa" exitCode=60 Oct 29 22:45:05.323217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.323117 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerDied","Data":"acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa"} Oct 29 22:45:05.323345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.323330 4543 scope.go:122] "RemoveContainer" containerID="acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa" Oct 29 22:45:05.323522 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.323503 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw_odh-ai-gateway-infra(cac5cbce-f0b2-4642-980f-c918af7c0747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" Oct 29 22:45:05.325089 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.325061 4543 scope.go:122] "RemoveContainer" containerID="07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19" Oct 29 22:45:05.325297 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.325261 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf_odh-ai-gateway-infra(73b34944-ce0b-4b0f-aee5-06d2b3559817)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" Oct 29 22:45:05.328155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.328135 4543 generic.go:356] "Generic (PLEG): container finished" podID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerID="f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2" exitCode=60 Oct 29 22:45:05.328261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.328175 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerDied","Data":"f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2"} Oct 29 22:45:05.328442 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.328422 4543 scope.go:122] "RemoveContainer" containerID="f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2" Oct 29 22:45:05.328693 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:05.328671 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7_odh-ai-gateway-infra(1ceda9a6-192b-4e99-ba66-2528b3161976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" Oct 29 22:45:05.331588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.331569 4543 scope.go:122] "RemoveContainer" containerID="98cd0cefc9124c393b099cec48841cde07c984457a254744d4e14c156077d443" Oct 29 22:45:05.344972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.344956 4543 scope.go:122] "RemoveContainer" containerID="fe03cf012bf636e5d6d6d9160a90c6e96e6159967afe5945d85f2424eedcc94b" Oct 29 22:45:05.358587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.358569 4543 scope.go:122] "RemoveContainer" containerID="03413d9233eb7b28d93b3c0f6d3a8f43dc650b6201d27c76f602f516e9c6fb85" Oct 29 22:45:05.372177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.372157 4543 scope.go:122] "RemoveContainer" containerID="86575e0e9f707b33db6d0b71ab87bf1e544be8bbe2d5a43928559e0447c4ed3b" Oct 29 22:45:05.444100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.443192 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" podStartSLOduration=2.443172242 podStartE2EDuration="2.443172242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:05.405493431 +0000 UTC m=+3767.147424105" watchObservedRunningTime="2026-10-29 22:45:05.443172242 +0000 UTC m=+3767.185102917" Oct 29 22:45:05.941681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.941608 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc"] Oct 29 22:45:05.946541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:05.946516 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d8760-fq9qc"] Oct 29 22:45:06.338611 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.338579 4543 scope.go:122] "RemoveContainer" containerID="f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2" Oct 29 22:45:06.339031 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:06.338867 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7_odh-ai-gateway-infra(1ceda9a6-192b-4e99-ba66-2528b3161976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" Oct 29 22:45:06.340158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.340130 4543 generic.go:356] "Generic (PLEG): container finished" podID="d8630ee4-7d00-4f36-95a4-634e3127c263" containerID="213ef68f1b3617114ed30b706de8a86ccb972be98e0124bb2281ba4dec1ea1a7" exitCode=0 Oct 29 22:45:06.340278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.340209 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" event={"ID":"d8630ee4-7d00-4f36-95a4-634e3127c263","Type":"ContainerDied","Data":"213ef68f1b3617114ed30b706de8a86ccb972be98e0124bb2281ba4dec1ea1a7"} Oct 29 22:45:06.342124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.342106 4543 scope.go:122] "RemoveContainer" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" Oct 29 22:45:06.342391 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:06.342371 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b_odh-ai-gateway-infra(98356f37-0e49-4513-99a2-d2f7c7f85ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" Oct 29 22:45:06.347576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.344175 4543 scope.go:122] "RemoveContainer" containerID="f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70" Oct 29 22:45:06.347576 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:06.344508 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl_odh-ai-gateway-infra(084b405f-b470-4fb4-99e8-a99bddd9bc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" Oct 29 22:45:06.349693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.349675 4543 scope.go:122] "RemoveContainer" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" Oct 29 22:45:06.349926 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:06.349905 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm_odh-ai-gateway-infra(ffcc7020-9571-4469-bc7c-1f496ac7a0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" Oct 29 22:45:06.351497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.351481 4543 scope.go:122] "RemoveContainer" containerID="acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa" Oct 29 22:45:06.351724 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:06.351703 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw_odh-ai-gateway-infra(cac5cbce-f0b2-4642-980f-c918af7c0747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" Oct 29 22:45:06.387028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:06.387004 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a713914-26c2-4d10-8a45-dbd0b2b2ce17" path="/var/lib/kubelet/pods/4a713914-26c2-4d10-8a45-dbd0b2b2ce17/volumes" Oct 29 22:45:07.519512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.519481 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:07.539195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.539173 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\") pod \"d8630ee4-7d00-4f36-95a4-634e3127c263\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " Oct 29 22:45:07.539342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.539292 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\" (UniqueName: \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\") pod \"d8630ee4-7d00-4f36-95a4-634e3127c263\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " Oct 29 22:45:07.539412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.539355 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\") pod \"d8630ee4-7d00-4f36-95a4-634e3127c263\" (UID: \"d8630ee4-7d00-4f36-95a4-634e3127c263\") " Oct 29 22:45:07.539566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.539543 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca" pod "d8630ee4-7d00-4f36-95a4-634e3127c263" (UID: "d8630ee4-7d00-4f36-95a4-634e3127c263"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:07.539905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.539883 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:07.541451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.541430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx" pod "d8630ee4-7d00-4f36-95a4-634e3127c263" (UID: "d8630ee4-7d00-4f36-95a4-634e3127c263"). InnerVolumeSpecName "kube-api-access-ldkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:07.541535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.541515 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca" pod "d8630ee4-7d00-4f36-95a4-634e3127c263" (UID: "d8630ee4-7d00-4f36-95a4-634e3127c263"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:07.640762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.640685 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldkdx\" (UniqueName: \"kubernetes.io/projected/d8630ee4-7d00-4f36-95a4-634e3127c263-kube-api-access-ldkdx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:07.640762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:07.640718 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8630ee4-7d00-4f36-95a4-634e3127c263-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:08.372776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:08.372750 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" Oct 29 22:45:08.372962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:08.372778 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s" event={"ID":"d8630ee4-7d00-4f36-95a4-634e3127c263","Type":"ContainerDied","Data":"782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e"} Oct 29 22:45:08.372962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:08.372824 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782a14be67b357ae5a4e66a299787138ed01fc7cacb965a3be4e69dddd98b65e" Oct 29 22:45:12.122835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:12.122800 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:12.123220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:12.122836 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:45:13.631257 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:13.631225 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:45:13.633321 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:13.631429 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" containerID="cri-o://a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111" gracePeriod=30 Oct 29 22:45:14.377989 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:14.377959 4543 scope.go:122] "RemoveContainer" containerID="26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4" Oct 29 22:45:14.378187 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:14.378040 4543 scope.go:122] "RemoveContainer" containerID="5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e" Oct 29 22:45:15.378716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.378688 4543 scope.go:122] "RemoveContainer" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" Oct 29 22:45:15.379139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.378968 4543 scope.go:122] "RemoveContainer" containerID="07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19" Oct 29 22:45:15.433658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.433629 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerStarted","Data":"5f2c67fd33a43bca993795b09f4efe9767c12806e4fc5be96fab490d46c95799"} Oct 29 22:45:15.435555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.435529 4543 generic.go:356] "Generic (PLEG): container finished" podID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerID="8ce236e61a16ecbdd86e5966ca013693cd3dde752a9c8d8a8704f565ddf4c18c" exitCode=60 Oct 29 22:45:15.435656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.435595 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerDied","Data":"8ce236e61a16ecbdd86e5966ca013693cd3dde752a9c8d8a8704f565ddf4c18c"} Oct 29 22:45:15.435656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.435650 4543 scope.go:122] "RemoveContainer" containerID="5f606358a4365cef4e9c8807aa11f417487846a72408f20431c722e6a612cd1e" Oct 29 22:45:15.436018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.435996 4543 scope.go:122] "RemoveContainer" containerID="8ce236e61a16ecbdd86e5966ca013693cd3dde752a9c8d8a8704f565ddf4c18c" Oct 29 22:45:15.436365 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:15.436338 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr_odh-ai-gateway-infra(d3d5daf2-a81e-4d8c-beb1-3ced71afc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" Oct 29 22:45:15.478825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:15.478781 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" podStartSLOduration=15.478768708 podStartE2EDuration="15.478768708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:15.477633353 +0000 UTC m=+3777.219564025" watchObservedRunningTime="2026-10-29 22:45:15.478768708 +0000 UTC m=+3777.220699381" Oct 29 22:45:16.378498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.378474 4543 scope.go:122] "RemoveContainer" containerID="f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70" Oct 29 22:45:16.378701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.378683 4543 scope.go:122] "RemoveContainer" containerID="f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2" Oct 29 22:45:16.378841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.378821 4543 scope.go:122] "RemoveContainer" containerID="acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa" Oct 29 22:45:16.447138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.447112 4543 generic.go:356] "Generic (PLEG): container finished" podID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerID="07bc449600fd63c46d59152ae5c2325db5e0460c264f18866cab127b70331b93" exitCode=60 Oct 29 22:45:16.447264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.447185 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerDied","Data":"07bc449600fd63c46d59152ae5c2325db5e0460c264f18866cab127b70331b93"} Oct 29 22:45:16.447264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.447241 4543 scope.go:122] "RemoveContainer" containerID="ed3beb752c937673bad0afae0ab810abea4b7c2770c6474c7d7bf116eed7e59a" Oct 29 22:45:16.447629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.447611 4543 scope.go:122] "RemoveContainer" containerID="07bc449600fd63c46d59152ae5c2325db5e0460c264f18866cab127b70331b93" Oct 29 22:45:16.447915 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:16.447893 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm_odh-ai-gateway-infra(ffcc7020-9571-4469-bc7c-1f496ac7a0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" Oct 29 22:45:16.449748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.449726 4543 generic.go:356] "Generic (PLEG): container finished" podID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerID="b318713493209cd427f465fc84d3b8f6dd14d5d141fd228cf70a90b3eacafe47" exitCode=60 Oct 29 22:45:16.449832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.449802 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerDied","Data":"b318713493209cd427f465fc84d3b8f6dd14d5d141fd228cf70a90b3eacafe47"} Oct 29 22:45:16.450434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.450202 4543 scope.go:122] "RemoveContainer" containerID="b318713493209cd427f465fc84d3b8f6dd14d5d141fd228cf70a90b3eacafe47" Oct 29 22:45:16.450500 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:16.450481 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf_odh-ai-gateway-infra(73b34944-ce0b-4b0f-aee5-06d2b3559817)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" Oct 29 22:45:16.452210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.452190 4543 generic.go:356] "Generic (PLEG): container finished" podID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerID="5f2c67fd33a43bca993795b09f4efe9767c12806e4fc5be96fab490d46c95799" exitCode=60 Oct 29 22:45:16.452295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.452267 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerDied","Data":"5f2c67fd33a43bca993795b09f4efe9767c12806e4fc5be96fab490d46c95799"} Oct 29 22:45:16.452860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.452651 4543 scope.go:122] "RemoveContainer" containerID="5f2c67fd33a43bca993795b09f4efe9767c12806e4fc5be96fab490d46c95799" Oct 29 22:45:16.452985 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:16.452961 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8_odh-ai-gateway-infra(39f2cd4c-eb99-4326-944d-a24aa7385d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" Oct 29 22:45:16.491761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.491738 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:16.521114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.520021 4543 scope.go:122] "RemoveContainer" containerID="07637306f06afe7be08ed328f80a370d1bf386a1b75efd83c9bcc88aec86ac19" Oct 29 22:45:16.534035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.534010 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:16.536978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.536959 4543 scope.go:122] "RemoveContainer" containerID="26e824043e5e86ac4b55d7b1890f3fe58e0617f97ac87c30304281e8bb03fbe4" Oct 29 22:45:16.573179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.573158 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:45:16.573360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.573334 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" containerID="cri-o://b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4" gracePeriod=30 Oct 29 22:45:16.887874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:16.887853 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:17.003687 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.003655 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\") pod \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " Oct 29 22:45:17.003881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.003863 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\" (UniqueName: \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\") pod \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\" (UID: \"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c\") " Oct 29 22:45:17.003991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.003969 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca" pod "d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" (UID: "d3d5daf2-a81e-4d8c-beb1-3ced71afc77c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:17.004403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.004382 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.006056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.006033 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc" pod "d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" (UID: "d3d5daf2-a81e-4d8c-beb1-3ced71afc77c"). InnerVolumeSpecName "kube-api-access-lldvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:17.105309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.105286 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lldvc\" (UniqueName: \"kubernetes.io/projected/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c-kube-api-access-lldvc\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.378675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.378649 4543 scope.go:122] "RemoveContainer" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" Oct 29 22:45:17.464813 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.464782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerStarted","Data":"3ec9e2f95c58126a00bbef91e2835a3fe32511faa51b8f7f329c7bff67ac24fe"} Oct 29 22:45:17.468625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.468601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerStarted","Data":"a8eb1b0a9dda695e597a8f57cbdf3a57232b9d84fb592eb4471b9686bd0b3dab"} Oct 29 22:45:17.472378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.472356 4543 generic.go:356] "Generic (PLEG): container finished" podID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerID="80d13a6a80d05741d519815d68320bdd30d364f2ec11665062efcbe1da60179a" exitCode=60 Oct 29 22:45:17.472491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.472431 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerDied","Data":"80d13a6a80d05741d519815d68320bdd30d364f2ec11665062efcbe1da60179a"} Oct 29 22:45:17.472557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.472495 4543 scope.go:122] "RemoveContainer" containerID="f06fec8afd958552eebe894ff956afad7e40942c577adfc40d8bb5979599c9c2" Oct 29 22:45:17.472879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.472851 4543 scope.go:122] "RemoveContainer" containerID="80d13a6a80d05741d519815d68320bdd30d364f2ec11665062efcbe1da60179a" Oct 29 22:45:17.473204 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:17.473154 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7_odh-ai-gateway-infra(1ceda9a6-192b-4e99-ba66-2528b3161976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" Oct 29 22:45:17.476202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.476182 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" event={"ID":"d3d5daf2-a81e-4d8c-beb1-3ced71afc77c","Type":"ContainerDied","Data":"2a7805724491d2ab64a25c7f4bd3d1a14998cbccd0340df365477fff2e5ce03a"} Oct 29 22:45:17.476202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.476199 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr" Oct 29 22:45:17.505655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.505635 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:17.601026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.598647 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:17.771380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.762593 4543 scope.go:122] "RemoveContainer" containerID="8ce236e61a16ecbdd86e5966ca013693cd3dde752a9c8d8a8704f565ddf4c18c" Oct 29 22:45:17.808865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.808842 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:17.815429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.815408 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:17.846022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.845369 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:17.857816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.857791 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-580af2-29888565-p2hfr"] Oct 29 22:45:17.869680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.869659 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.914441 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\" (UniqueName: \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\") pod \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.914646 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\") pod \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.914700 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\") pod \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\" (UID: \"ffcc7020-9571-4469-bc7c-1f496ac7a0f6\") " Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.914784 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\" (UniqueName: \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\") pod \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\" (UID: \"39f2cd4c-eb99-4326-944d-a24aa7385d8f\") " Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.915154 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca" pod "39f2cd4c-eb99-4326-944d-a24aa7385d8f" (UID: "39f2cd4c-eb99-4326-944d-a24aa7385d8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:17.915266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.915167 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca" pod "ffcc7020-9571-4469-bc7c-1f496ac7a0f6" (UID: "ffcc7020-9571-4469-bc7c-1f496ac7a0f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:17.915609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.915340 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f2cd4c-eb99-4326-944d-a24aa7385d8f-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.915609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.915357 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.917299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.917275 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z" pod "ffcc7020-9571-4469-bc7c-1f496ac7a0f6" (UID: "ffcc7020-9571-4469-bc7c-1f496ac7a0f6"). InnerVolumeSpecName "kube-api-access-5jr8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:17.917395 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:17.917309 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp" pod "39f2cd4c-eb99-4326-944d-a24aa7385d8f" (UID: "39f2cd4c-eb99-4326-944d-a24aa7385d8f"). InnerVolumeSpecName "kube-api-access-8h6gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.016554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.016513 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\" (UniqueName: \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\") pod \"73b34944-ce0b-4b0f-aee5-06d2b3559817\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " Oct 29 22:45:18.016795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.016773 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\") pod \"73b34944-ce0b-4b0f-aee5-06d2b3559817\" (UID: \"73b34944-ce0b-4b0f-aee5-06d2b3559817\") " Oct 29 22:45:18.017134 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.017108 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca" pod "73b34944-ce0b-4b0f-aee5-06d2b3559817" (UID: "73b34944-ce0b-4b0f-aee5-06d2b3559817"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.017372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.017354 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b34944-ce0b-4b0f-aee5-06d2b3559817-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.017454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.017373 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6gp\" (UniqueName: \"kubernetes.io/projected/39f2cd4c-eb99-4326-944d-a24aa7385d8f-kube-api-access-8h6gp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.017454 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.017384 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jr8z\" (UniqueName: \"kubernetes.io/projected/ffcc7020-9571-4469-bc7c-1f496ac7a0f6-kube-api-access-5jr8z\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.019387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.019358 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7" pod "73b34944-ce0b-4b0f-aee5-06d2b3559817" (UID: "73b34944-ce0b-4b0f-aee5-06d2b3559817"). InnerVolumeSpecName "kube-api-access-2vdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.118218 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.118188 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdf7\" (UniqueName: \"kubernetes.io/projected/73b34944-ce0b-4b0f-aee5-06d2b3559817-kube-api-access-2vdf7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.387032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.387002 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" path="/var/lib/kubelet/pods/d3d5daf2-a81e-4d8c-beb1-3ced71afc77c/volumes" Oct 29 22:45:18.486868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.486839 4543 generic.go:356] "Generic (PLEG): container finished" podID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerID="fe7aafa7f82a34ee475179b0a9ea2570070688aacbedf1b54c769cbc8af6d3f8" exitCode=60 Oct 29 22:45:18.487255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.486914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerDied","Data":"fe7aafa7f82a34ee475179b0a9ea2570070688aacbedf1b54c769cbc8af6d3f8"} Oct 29 22:45:18.487255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.486971 4543 scope.go:122] "RemoveContainer" containerID="8466d348de0dfb005ddb924484ee78f185f87cdf6f20654a3669960f905677e8" Oct 29 22:45:18.487380 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.487370 4543 scope.go:122] "RemoveContainer" containerID="fe7aafa7f82a34ee475179b0a9ea2570070688aacbedf1b54c769cbc8af6d3f8" Oct 29 22:45:18.487704 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:18.487680 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b_odh-ai-gateway-infra(98356f37-0e49-4513-99a2-d2f7c7f85ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" Oct 29 22:45:18.489224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.489200 4543 generic.go:356] "Generic (PLEG): container finished" podID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerID="fb7bc2b854462de810345e048b4588ac5d84a6cb23ddd64d822ec778e13e4779" exitCode=0 Oct 29 22:45:18.489329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.489282 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" event={"ID":"b450c07e-fd5b-486f-9f9e-d5e299b3aad5","Type":"ContainerDied","Data":"fb7bc2b854462de810345e048b4588ac5d84a6cb23ddd64d822ec778e13e4779"} Oct 29 22:45:18.491590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.491566 4543 generic.go:356] "Generic (PLEG): container finished" podID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerID="3ec9e2f95c58126a00bbef91e2835a3fe32511faa51b8f7f329c7bff67ac24fe" exitCode=60 Oct 29 22:45:18.491724 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.491595 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerDied","Data":"3ec9e2f95c58126a00bbef91e2835a3fe32511faa51b8f7f329c7bff67ac24fe"} Oct 29 22:45:18.492058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.491995 4543 scope.go:122] "RemoveContainer" containerID="3ec9e2f95c58126a00bbef91e2835a3fe32511faa51b8f7f329c7bff67ac24fe" Oct 29 22:45:18.492382 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:18.492355 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl_odh-ai-gateway-infra(084b405f-b470-4fb4-99e8-a99bddd9bc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" Oct 29 22:45:18.493638 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.493616 4543 generic.go:356] "Generic (PLEG): container finished" podID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerID="09b517d0c4ca8a301faebc3505b4f8e764db07ff63ed91d30febe49cde806a87" exitCode=0 Oct 29 22:45:18.493744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.493700 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" event={"ID":"d2036a9c-816c-45bf-b4d8-dc59815ff9f4","Type":"ContainerDied","Data":"09b517d0c4ca8a301faebc3505b4f8e764db07ff63ed91d30febe49cde806a87"} Oct 29 22:45:18.495282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.495263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" event={"ID":"ffcc7020-9571-4469-bc7c-1f496ac7a0f6","Type":"ContainerDied","Data":"dd2265b3e23a132f0c32a352663b95a34a8af65b7056e339d951d11a77258098"} Oct 29 22:45:18.495375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.495300 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm" Oct 29 22:45:18.502056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.502032 4543 generic.go:356] "Generic (PLEG): container finished" podID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerID="a8eb1b0a9dda695e597a8f57cbdf3a57232b9d84fb592eb4471b9686bd0b3dab" exitCode=60 Oct 29 22:45:18.502183 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.502148 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerDied","Data":"a8eb1b0a9dda695e597a8f57cbdf3a57232b9d84fb592eb4471b9686bd0b3dab"} Oct 29 22:45:18.502545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.502528 4543 scope.go:122] "RemoveContainer" containerID="a8eb1b0a9dda695e597a8f57cbdf3a57232b9d84fb592eb4471b9686bd0b3dab" Oct 29 22:45:18.502926 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:18.502901 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw_odh-ai-gateway-infra(cac5cbce-f0b2-4642-980f-c918af7c0747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" Oct 29 22:45:18.507476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.505726 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" event={"ID":"73b34944-ce0b-4b0f-aee5-06d2b3559817","Type":"ContainerDied","Data":"388176f700f68f19d77be8b287d2392e50e96a849451eb5a5686a41d0f094a9f"} Oct 29 22:45:18.507476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.505814 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf" Oct 29 22:45:18.512589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.512401 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" Oct 29 22:45:18.512589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.512426 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8" event={"ID":"39f2cd4c-eb99-4326-944d-a24aa7385d8f","Type":"ContainerDied","Data":"836fc0c862e224e87dc26b9b99c798655d28845dc11d78d0370a0b5d58b5c013"} Oct 29 22:45:18.519322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.519281 4543 scope.go:122] "RemoveContainer" containerID="f8e6906fb95aae647e045ddc9e4792d93dedf8fb321da38b2b2d5541cd8fac70" Oct 29 22:45:18.524624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.524539 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:18.537008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.536989 4543 scope.go:122] "RemoveContainer" containerID="07bc449600fd63c46d59152ae5c2325db5e0460c264f18866cab127b70331b93" Oct 29 22:45:18.550958 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.550934 4543 scope.go:122] "RemoveContainer" containerID="acdccd2441ee77d54536dd3b4823d576686c6b8a510bf08c22ad062de6c51aaa" Oct 29 22:45:18.566449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.566395 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:18.582026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.582003 4543 scope.go:122] "RemoveContainer" containerID="b318713493209cd427f465fc84d3b8f6dd14d5d141fd228cf70a90b3eacafe47" Oct 29 22:45:18.601891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.600847 4543 scope.go:122] "RemoveContainer" containerID="5f2c67fd33a43bca993795b09f4efe9767c12806e4fc5be96fab490d46c95799" Oct 29 22:45:18.664983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.664957 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:18.670309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.670281 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-27b6db-29888565-txjpf"] Oct 29 22:45:18.713278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.713252 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:18.719509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.719485 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-563f39-29888565-bwznm"] Oct 29 22:45:18.749491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.749467 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:45:18.753240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.753220 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:45:18.753331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.753310 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:18.758127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.758105 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:18.766995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.766973 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4afaba-29888565-jd6q8"] Oct 29 22:45:18.828709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.828673 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\" (UniqueName: \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\") pod \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " Oct 29 22:45:18.828841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.828750 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\") pod \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\" (UID: \"b450c07e-fd5b-486f-9f9e-d5e299b3aad5\") " Oct 29 22:45:18.828945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.828922 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\") pod \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " Oct 29 22:45:18.829035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.829012 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\" (UniqueName: \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\") pod \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\" (UID: \"d2036a9c-816c-45bf-b4d8-dc59815ff9f4\") " Oct 29 22:45:18.831207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.831175 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume" pod "b450c07e-fd5b-486f-9f9e-d5e299b3aad5" (UID: "b450c07e-fd5b-486f-9f9e-d5e299b3aad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.831593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.831567 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume" pod "d2036a9c-816c-45bf-b4d8-dc59815ff9f4" (UID: "d2036a9c-816c-45bf-b4d8-dc59815ff9f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.834455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.834421 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn" pod "b450c07e-fd5b-486f-9f9e-d5e299b3aad5" (UID: "b450c07e-fd5b-486f-9f9e-d5e299b3aad5"). InnerVolumeSpecName "kube-api-access-4kfgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.840441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.840411 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms" pod "d2036a9c-816c-45bf-b4d8-dc59815ff9f4" (UID: "d2036a9c-816c-45bf-b4d8-dc59815ff9f4"). InnerVolumeSpecName "kube-api-access-4dtms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.916765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.916733 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:18.931364 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.931334 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kfgn\" (UniqueName: \"kubernetes.io/projected/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-kube-api-access-4kfgn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.931533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.931368 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b450c07e-fd5b-486f-9f9e-d5e299b3aad5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.931533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.931385 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.931533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.931401 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtms\" (UniqueName: \"kubernetes.io/projected/d2036a9c-816c-45bf-b4d8-dc59815ff9f4-kube-api-access-4dtms\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.969042 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:18.969015 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:45:19.032147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032118 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032170 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032259 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\") pod \"1ceda9a6-192b-4e99-ba66-2528b3161976\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " Oct 29 22:45:19.032299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032283 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032310 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\" (UniqueName: \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\") pod \"1ceda9a6-192b-4e99-ba66-2528b3161976\" (UID: \"1ceda9a6-192b-4e99-ba66-2528b3161976\") " Oct 29 22:45:19.032483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032382 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032478 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032607 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032532 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032605 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032657 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032698 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\") pod \"176acca7-2dee-4eb0-9791-7f5ac960013f\" (UID: \"176acca7-2dee-4eb0-9791-7f5ac960013f\") " Oct 29 22:45:19.032729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.032696 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca" pod "1ceda9a6-192b-4e99-ba66-2528b3161976" (UID: "1ceda9a6-192b-4e99-ba66-2528b3161976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.033185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033130 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.033523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033379 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:19.033523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033010 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:19.033632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033608 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:19.033669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033632 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ceda9a6-192b-4e99-ba66-2528b3161976-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.033669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033651 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.033740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033667 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176acca7-2dee-4eb0-9791-7f5ac960013f-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.033740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033682 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.033858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.033836 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:19.034929 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.034903 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "kube-api-access-nhsx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.035126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.035068 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:19.035126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.035112 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.035694 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.035675 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5" pod "1ceda9a6-192b-4e99-ba66-2528b3161976" (UID: "1ceda9a6-192b-4e99-ba66-2528b3161976"). InnerVolumeSpecName "kube-api-access-rswz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.035987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.035957 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo" pod "176acca7-2dee-4eb0-9791-7f5ac960013f" (UID: "176acca7-2dee-4eb0-9791-7f5ac960013f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135195 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135231 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135245 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135258 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135270 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135283 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsx2\" (UniqueName: \"kubernetes.io/projected/176acca7-2dee-4eb0-9791-7f5ac960013f-kube-api-access-nhsx2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.135630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.135297 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rswz5\" (UniqueName: \"kubernetes.io/projected/1ceda9a6-192b-4e99-ba66-2528b3161976-kube-api-access-rswz5\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.528330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.528294 4543 generic.go:356] "Generic (PLEG): container finished" podID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerID="a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111" exitCode=0 Oct 29 22:45:19.528778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.528354 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" Oct 29 22:45:19.528778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.528386 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerDied","Data":"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111"} Oct 29 22:45:19.528778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.528440 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p" event={"ID":"176acca7-2dee-4eb0-9791-7f5ac960013f","Type":"ContainerDied","Data":"4e2c95aedac732981ac6f0d2e8f5a6d5f7507232697a40eb2518b2bebd3a1636"} Oct 29 22:45:19.528778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.528463 4543 scope.go:122] "RemoveContainer" containerID="a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111" Oct 29 22:45:19.529966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.529950 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" Oct 29 22:45:19.530067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.529972 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv" event={"ID":"b450c07e-fd5b-486f-9f9e-d5e299b3aad5","Type":"ContainerDied","Data":"f22e74bee3e35d678478e831b1222ce38bdada11f26b65e4774a2063058de48d"} Oct 29 22:45:19.533745 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.533723 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" Oct 29 22:45:19.533840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.533756 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8" event={"ID":"d2036a9c-816c-45bf-b4d8-dc59815ff9f4","Type":"ContainerDied","Data":"4f6d4119cf14f3a06152e71f01fd3c8226c91d890d271b49ada92e107ca24706"} Oct 29 22:45:19.551438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.551413 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:19.556705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.556671 4543 scope.go:122] "RemoveContainer" containerID="15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a" Oct 29 22:45:19.557992 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.557895 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" event={"ID":"1ceda9a6-192b-4e99-ba66-2528b3161976","Type":"ContainerDied","Data":"209a3c6f74636b72d6d55ed36d7189db8dcf47c26c4d479d240d0915a93f9ba7"} Oct 29 22:45:19.558155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.558034 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7" Oct 29 22:45:19.605503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.605471 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:45:19.605643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.605520 4543 scope.go:122] "RemoveContainer" containerID="a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111" Oct 29 22:45:19.605867 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:19.605838 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111\": container with ID starting with a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111 not found: ID does not exist" containerID="a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111" Oct 29 22:45:19.605978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.605877 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111"} err="failed to get container status \"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111\": rpc error: code = NotFound desc = could not find container \"a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111\": container with ID starting with a193a72f042d4864f554c777d3bc5b6f7c66f5155b7babcdef0a18a154dcf111 not found: ID does not exist" Oct 29 22:45:19.605978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.605901 4543 scope.go:122] "RemoveContainer" containerID="15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a" Oct 29 22:45:19.607256 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:19.607225 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a\": container with ID starting with 15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a not found: ID does not exist" containerID="15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a" Oct 29 22:45:19.607362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.607265 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a"} err="failed to get container status \"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a\": rpc error: code = NotFound desc = could not find container \"15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a\": container with ID starting with 15c10abc8b07ffa381b62185f64a46c9be714a7265acf86531db4c5026ef5d0a not found: ID does not exist" Oct 29 22:45:19.607362 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.607285 4543 scope.go:122] "RemoveContainer" containerID="fb7bc2b854462de810345e048b4588ac5d84a6cb23ddd64d822ec778e13e4779" Oct 29 22:45:19.610366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.609663 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d8760-openshift-default-8dd555fc-btl5p"] Oct 29 22:45:19.623698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.623669 4543 scope.go:122] "RemoveContainer" containerID="09b517d0c4ca8a301faebc3505b4f8e764db07ff63ed91d30febe49cde806a87" Oct 29 22:45:19.641960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.640493 4543 scope.go:122] "RemoveContainer" containerID="80d13a6a80d05741d519815d68320bdd30d364f2ec11665062efcbe1da60179a" Oct 29 22:45:19.822224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.822172 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:45:19.825404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.825328 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:19.825720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.825703 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d8760-bbdf99759-nvdl8"] Oct 29 22:45:19.850884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.850853 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\") pod \"084b405f-b470-4fb4-99e8-a99bddd9bc56\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " Oct 29 22:45:19.851014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.850979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\" (UniqueName: \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\") pod \"084b405f-b470-4fb4-99e8-a99bddd9bc56\" (UID: \"084b405f-b470-4fb4-99e8-a99bddd9bc56\") " Oct 29 22:45:19.851422 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.851378 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca" pod "084b405f-b470-4fb4-99e8-a99bddd9bc56" (UID: "084b405f-b470-4fb4-99e8-a99bddd9bc56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.862921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.862734 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp" pod "084b405f-b470-4fb4-99e8-a99bddd9bc56" (UID: "084b405f-b470-4fb4-99e8-a99bddd9bc56"). InnerVolumeSpecName "kube-api-access-fd6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.867164 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.867138 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc"] Oct 29 22:45:19.868299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868274 4543 cpu_manager.go:414] "containerMap: removing container" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868301 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868329 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868338 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868351 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868358 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868370 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868377 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868390 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868396 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868414 4543 cpu_manager.go:414] "containerMap: removing container" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868420 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868438 4543 cpu_manager.go:414] "containerMap: removing container" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.868448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868445 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868458 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868465 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868477 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868483 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868496 4543 cpu_manager.go:414] "containerMap: removing container" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868503 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868513 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868520 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868542 4543 cpu_manager.go:414] "containerMap: removing container" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868549 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868565 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868571 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868583 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d8630ee4-7d00-4f36-95a4-634e3127c263" containerName="revoke-keys" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868590 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8630ee4-7d00-4f36-95a4-634e3127c263" containerName="revoke-keys" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868608 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868615 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868627 4543 cpu_manager.go:414] "containerMap: removing container" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868633 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868651 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868657 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868821 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868833 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868841 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868852 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868863 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868870 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868879 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868890 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868899 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868915 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868925 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868936 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868944 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868953 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" containerName="payload-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868960 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8630ee4-7d00-4f36-95a4-634e3127c263" containerName="revoke-keys" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868968 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868976 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" containerName="payload-pre-processing" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868983 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" containerName="istio-proxy" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.868993 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:19.869341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.869001 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:19.873968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.873948 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:19.890120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.890096 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:19.893679 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.893653 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc"] Oct 29 22:45:19.899504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.899478 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:19.910353 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.910328 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:19.914159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.914141 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888565-krsd7"] Oct 29 22:45:19.915780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.915760 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:19.952014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.951990 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\" (UniqueName: \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\") pod \"cac5cbce-f0b2-4642-980f-c918af7c0747\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " Oct 29 22:45:19.952227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952203 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\" (UniqueName: \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\") pod \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " Oct 29 22:45:19.952322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952253 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\") pod \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\" (UID: \"98356f37-0e49-4513-99a2-d2f7c7f85ca7\") " Oct 29 22:45:19.952487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\") pod \"cac5cbce-f0b2-4642-980f-c918af7c0747\" (UID: \"cac5cbce-f0b2-4642-980f-c918af7c0747\") " Oct 29 22:45:19.952722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952697 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca" pod "98356f37-0e49-4513-99a2-d2f7c7f85ca7" (UID: "98356f37-0e49-4513-99a2-d2f7c7f85ca7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.952814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952795 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5s98\" (UniqueName: \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:19.952890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.952876 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:19.953118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953058 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:19.953118 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca" pod "cac5cbce-f0b2-4642-980f-c918af7c0747" (UID: "cac5cbce-f0b2-4642-980f-c918af7c0747"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.953330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953240 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98356f37-0e49-4513-99a2-d2f7c7f85ca7-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.953330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953261 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6wp\" (UniqueName: \"kubernetes.io/projected/084b405f-b470-4fb4-99e8-a99bddd9bc56-kube-api-access-fd6wp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.953330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953278 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cbce-f0b2-4642-980f-c918af7c0747-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.953330 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.953295 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084b405f-b470-4fb4-99e8-a99bddd9bc56-maas-api-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.954598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.954570 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w" pod "cac5cbce-f0b2-4642-980f-c918af7c0747" (UID: "cac5cbce-f0b2-4642-980f-c918af7c0747"). InnerVolumeSpecName "kube-api-access-mv84w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.955184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.955019 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t" pod "98356f37-0e49-4513-99a2-d2f7c7f85ca7" (UID: "98356f37-0e49-4513-99a2-d2f7c7f85ca7"). InnerVolumeSpecName "kube-api-access-v9s7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.956377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.956354 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:45:19.959715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:19.959693 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d8760-8968b48c8-75vhv"] Oct 29 22:45:20.054146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.054297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054215 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.054436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5s98\" (UniqueName: \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.054550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054522 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv84w\" (UniqueName: \"kubernetes.io/projected/cac5cbce-f0b2-4642-980f-c918af7c0747-kube-api-access-mv84w\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.054792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054767 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9s7t\" (UniqueName: \"kubernetes.io/projected/98356f37-0e49-4513-99a2-d2f7c7f85ca7-kube-api-access-v9s7t\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.054883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.054828 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.057517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.057485 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.081108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.081067 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5s98\" (UniqueName: \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\") pod \"maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.214262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.214232 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:20.386035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.385959 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176acca7-2dee-4eb0-9791-7f5ac960013f" path="/var/lib/kubelet/pods/176acca7-2dee-4eb0-9791-7f5ac960013f/volumes" Oct 29 22:45:20.386413 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.386398 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" path="/var/lib/kubelet/pods/1ceda9a6-192b-4e99-ba66-2528b3161976/volumes" Oct 29 22:45:20.386778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.386765 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" path="/var/lib/kubelet/pods/39f2cd4c-eb99-4326-944d-a24aa7385d8f/volumes" Oct 29 22:45:20.387459 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.387442 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" path="/var/lib/kubelet/pods/73b34944-ce0b-4b0f-aee5-06d2b3559817/volumes" Oct 29 22:45:20.387817 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.387805 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b450c07e-fd5b-486f-9f9e-d5e299b3aad5" path="/var/lib/kubelet/pods/b450c07e-fd5b-486f-9f9e-d5e299b3aad5/volumes" Oct 29 22:45:20.388196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.388182 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2036a9c-816c-45bf-b4d8-dc59815ff9f4" path="/var/lib/kubelet/pods/d2036a9c-816c-45bf-b4d8-dc59815ff9f4/volumes" Oct 29 22:45:20.389020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.388998 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" path="/var/lib/kubelet/pods/ffcc7020-9571-4469-bc7c-1f496ac7a0f6/volumes" Oct 29 22:45:20.537951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.537928 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc"] Oct 29 22:45:20.538747 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:20.538719 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d88eb8c_5b74_432b_9ecb_0d5c755a565b.slice/crio-63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7 WatchSource:0}: Error finding container 63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7: Status 404 returned error can't find the container with id 63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7 Oct 29 22:45:20.571172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.571136 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" Oct 29 22:45:20.571289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.571167 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b" event={"ID":"98356f37-0e49-4513-99a2-d2f7c7f85ca7","Type":"ContainerDied","Data":"9fc145b206abff59e2e6f9274d53b79d9446b5b57672838f18ba8b6de3dae722"} Oct 29 22:45:20.571289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.571245 4543 scope.go:122] "RemoveContainer" containerID="fe7aafa7f82a34ee475179b0a9ea2570070688aacbedf1b54c769cbc8af6d3f8" Oct 29 22:45:20.576723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.576243 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" event={"ID":"084b405f-b470-4fb4-99e8-a99bddd9bc56","Type":"ContainerDied","Data":"2c0c2e0b600d1f17d5ca8b1089a412f2127dd9242b984aec3a9560defa35aefb"} Oct 29 22:45:20.576723 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.576252 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl" Oct 29 22:45:20.578836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.578816 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" Oct 29 22:45:20.578972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.578859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw" event={"ID":"cac5cbce-f0b2-4642-980f-c918af7c0747","Type":"ContainerDied","Data":"3d38eaca6ea635266f0a17190ddee4699c141e82e8c99fee0aba32044f5f1bea"} Oct 29 22:45:20.582019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.581996 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" event={"ID":"0d88eb8c-5b74-432b-9ecb-0d5c755a565b","Type":"ContainerStarted","Data":"63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7"} Oct 29 22:45:20.587297 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.587279 4543 scope.go:122] "RemoveContainer" containerID="3ec9e2f95c58126a00bbef91e2835a3fe32511faa51b8f7f329c7bff67ac24fe" Oct 29 22:45:20.596761 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.596741 4543 scope.go:122] "RemoveContainer" containerID="a8eb1b0a9dda695e597a8f57cbdf3a57232b9d84fb592eb4471b9686bd0b3dab" Oct 29 22:45:20.629779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.629756 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:20.634495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.634477 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888565-n8vcl"] Oct 29 22:45:20.673172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.673120 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:20.681545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.681526 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888565-gr28b"] Oct 29 22:45:20.721282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.721262 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:20.732221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:20.732204 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888565-ds5bw"] Oct 29 22:45:21.596314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.596283 4543 generic.go:356] "Generic (PLEG): container finished" podID="0d88eb8c-5b74-432b-9ecb-0d5c755a565b" containerID="fefd7c780d3dcd665cece19c0ccb616f480111ebd3a35acc72be8f3caa9e4b48" exitCode=0 Oct 29 22:45:21.596677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.596364 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" event={"ID":"0d88eb8c-5b74-432b-9ecb-0d5c755a565b","Type":"ContainerDied","Data":"fefd7c780d3dcd665cece19c0ccb616f480111ebd3a35acc72be8f3caa9e4b48"} Oct 29 22:45:21.838467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.838396 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:45:21.873833 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.873811 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.873970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.873898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.873970 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.873947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874099 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:21.874129 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874108 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874202 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874234 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874212 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:21.874348 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874316 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.874397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874372 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874449 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874447 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874505 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874479 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\") pod \"ff40db28-0d24-4cca-a739-5d0c3448503e\" (UID: \"ff40db28-0d24-4cca-a739-5d0c3448503e\") " Oct 29 22:45:21.874803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.874759 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:21.875123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.875101 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.875123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.875124 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff40db28-0d24-4cca-a739-5d0c3448503e-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.875290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.875133 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.875290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.875142 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.875424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.875353 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:21.876527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.876500 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:21.876953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.876928 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "kube-api-access-knwmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.877545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.877524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:21.877644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.877564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token" pod "ff40db28-0d24-4cca-a739-5d0c3448503e" (UID: "ff40db28-0d24-4cca-a739-5d0c3448503e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.976680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.976652 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.976680 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.976678 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.976862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.976688 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwmj\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-kube-api-access-knwmj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.976862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.976698 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff40db28-0d24-4cca-a739-5d0c3448503e-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.976862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:21.976707 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff40db28-0d24-4cca-a739-5d0c3448503e-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.385672 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.385596 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" path="/var/lib/kubelet/pods/084b405f-b470-4fb4-99e8-a99bddd9bc56/volumes" Oct 29 22:45:22.386029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.386015 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" path="/var/lib/kubelet/pods/98356f37-0e49-4513-99a2-d2f7c7f85ca7/volumes" Oct 29 22:45:22.386465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.386451 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" path="/var/lib/kubelet/pods/cac5cbce-f0b2-4642-980f-c918af7c0747/volumes" Oct 29 22:45:22.606549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.606522 4543 generic.go:356] "Generic (PLEG): container finished" podID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerID="b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4" exitCode=0 Oct 29 22:45:22.607015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.606569 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" Oct 29 22:45:22.607015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.606609 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" event={"ID":"ff40db28-0d24-4cca-a739-5d0c3448503e","Type":"ContainerDied","Data":"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4"} Oct 29 22:45:22.607015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.606666 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q" event={"ID":"ff40db28-0d24-4cca-a739-5d0c3448503e","Type":"ContainerDied","Data":"dcb62d5d65a9bf8c2ab71aa9c2eef24154b68606872a0db68c5f8dac624ae5c5"} Oct 29 22:45:22.607015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.606694 4543 scope.go:122] "RemoveContainer" containerID="b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4" Oct 29 22:45:22.617799 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.617779 4543 scope.go:122] "RemoveContainer" containerID="b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4" Oct 29 22:45:22.618113 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:22.618095 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4\": container with ID starting with b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4 not found: ID does not exist" containerID="b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4" Oct 29 22:45:22.618174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.618119 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4"} err="failed to get container status \"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4\": rpc error: code = NotFound desc = could not find container \"b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4\": container with ID starting with b29961c2667cf04756be79918c55ff65eecb90d5db25290bd249a3e0d9c4f4f4 not found: ID does not exist" Oct 29 22:45:22.681335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.681307 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:45:22.694252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.694223 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dfq6q"] Oct 29 22:45:22.756878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.756855 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:22.787339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.787313 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\" (UniqueName: \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\") pod \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " Oct 29 22:45:22.787475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.787404 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\") pod \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " Oct 29 22:45:22.787553 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.787505 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\") pod \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\" (UID: \"0d88eb8c-5b74-432b-9ecb-0d5c755a565b\") " Oct 29 22:45:22.787879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.787854 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca" pod "0d88eb8c-5b74-432b-9ecb-0d5c755a565b" (UID: "0d88eb8c-5b74-432b-9ecb-0d5c755a565b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:22.788108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.788044 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.789495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.789474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca" pod "0d88eb8c-5b74-432b-9ecb-0d5c755a565b" (UID: "0d88eb8c-5b74-432b-9ecb-0d5c755a565b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:22.789604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.789580 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98" pod "0d88eb8c-5b74-432b-9ecb-0d5c755a565b" (UID: "0d88eb8c-5b74-432b-9ecb-0d5c755a565b"). InnerVolumeSpecName "kube-api-access-q5s98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:22.889710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.889650 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5s98\" (UniqueName: \"kubernetes.io/projected/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-kube-api-access-q5s98\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.889710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:22.889686 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb8c-5b74-432b-9ecb-0d5c755a565b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:23.616405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:23.616377 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" Oct 29 22:45:23.616815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:23.616377 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc" event={"ID":"0d88eb8c-5b74-432b-9ecb-0d5c755a565b","Type":"ContainerDied","Data":"63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7"} Oct 29 22:45:23.616815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:23.616492 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63fcd3658f17a97f1bdf7a469333bb9f4b2b2891fe74dab1c83de44ccc119fa7" Oct 29 22:45:24.386595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:24.386557 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" path="/var/lib/kubelet/pods/ff40db28-0d24-4cca-a739-5d0c3448503e/volumes" Oct 29 22:45:24.667881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:24.667803 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:45:24.708243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:24.708212 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:45:25.004678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:25.004651 4543 scope.go:122] "RemoveContainer" containerID="ffc31e6218fd25243af220741ebaa54fe5e80a4bcbf63d0cb5feb993a330260f" Oct 29 22:45:25.015278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:25.015258 4543 scope.go:122] "RemoveContainer" containerID="83621820aeabe4eb26e70d0e4c5e8bf09db9b80d726acf56aac5471749a71a82" Oct 29 22:45:25.024836 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:25.024816 4543 scope.go:122] "RemoveContainer" containerID="6d2aaead9e90254b7e29f21e61c4d552a36458275953b1387bf9380225c51ecc" Oct 29 22:45:25.034268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:25.034251 4543 scope.go:122] "RemoveContainer" containerID="cfbc235039840ffabe473f2bc3552e6598cc1f390d3d337eb85fe5cc8fcaeed7" Oct 29 22:45:27.962208 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:27.962161 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:45:28.670563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.670529 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f"] Oct 29 22:45:28.671124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671109 4543 cpu_manager.go:414] "containerMap: removing container" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671126 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b34944-ce0b-4b0f-aee5-06d2b3559817" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671137 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671141 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d5daf2-a81e-4d8c-beb1-3ced71afc77c" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671152 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671157 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671167 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671171 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671180 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671185 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671198 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671204 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671214 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671218 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671232 4543 cpu_manager.go:414] "containerMap: removing container" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671236 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671246 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671250 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671260 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0d88eb8c-5b74-432b-9ecb-0d5c755a565b" containerName="revoke-keys" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671264 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d88eb8c-5b74-432b-9ecb-0d5c755a565b" containerName="revoke-keys" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671271 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671275 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671280 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671285 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcc7020-9571-4469-bc7c-1f496ac7a0f6" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671369 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671379 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff40db28-0d24-4cca-a739-5d0c3448503e" containerName="istio-proxy" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671387 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d88eb8c-5b74-432b-9ecb-0d5c755a565b" containerName="revoke-keys" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671392 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f2cd4c-eb99-4326-944d-a24aa7385d8f" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671397 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671402 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="084b405f-b470-4fb4-99e8-a99bddd9bc56" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671412 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceda9a6-192b-4e99-ba66-2528b3161976" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671419 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:28.671476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.671426 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:28.677356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.677338 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.695590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.695569 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f"] Oct 29 22:45:28.704252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.704232 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:28.704252 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.704232 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:28.802923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.802896 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdr9\" (UniqueName: \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.803068 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.802960 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.803186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.803139 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.904311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.904286 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.904415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.904393 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdr9\" (UniqueName: \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.904478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.904432 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.905064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.905046 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.906719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.906698 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.930497 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.930449 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdr9\" (UniqueName: \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\") pod \"maas-api-revoke-sub-b713f27b-ptp8f\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:28.991020 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:28.991000 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:29.298222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.298183 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f"] Oct 29 22:45:29.299278 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:29.299252 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149ec293_c313_4360_b728_250da7da7388.slice/crio-bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51 WatchSource:0}: Error finding container bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51: Status 404 returned error can't find the container with id bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51 Oct 29 22:45:29.663262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.663234 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" event={"ID":"149ec293-c313-4360-b728-250da7da7388","Type":"ContainerStarted","Data":"97a9f264ce06d9fc76d2fb876e2821669d8fe258bdb5fe34f97ee6d17ce8f4f1"} Oct 29 22:45:29.663375 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.663286 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" event={"ID":"149ec293-c313-4360-b728-250da7da7388","Type":"ContainerStarted","Data":"bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51"} Oct 29 22:45:29.668499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.668474 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" podUID="78493013-3db7-46a0-a3c0-bce2b9050541" containerName="payload-processing" containerID="cri-o://d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273" gracePeriod=25 Oct 29 22:45:29.712224 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.712170 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" podStartSLOduration=1.712152793 podStartE2EDuration="1.712152793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:29.705588933 +0000 UTC m=+3791.447519606" watchObservedRunningTime="2026-10-29 22:45:29.712152793 +0000 UTC m=+3791.454083467" Oct 29 22:45:29.714180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.714138 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" podUID="a2be3143-8369-43b8-83e3-99e21b36903d" containerName="payload-pre-processing" containerID="cri-o://cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6" gracePeriod=25 Oct 29 22:45:29.982065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:29.982038 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:45:30.020976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.020956 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:45:30.132589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.132556 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\" (UniqueName: \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\") pod \"a2be3143-8369-43b8-83e3-99e21b36903d\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " Oct 29 22:45:30.132751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.132617 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\") pod \"78493013-3db7-46a0-a3c0-bce2b9050541\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " Oct 29 22:45:30.132751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.132669 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\") pod \"a2be3143-8369-43b8-83e3-99e21b36903d\" (UID: \"a2be3143-8369-43b8-83e3-99e21b36903d\") " Oct 29 22:45:30.132890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.132765 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\" (UniqueName: \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\") pod \"78493013-3db7-46a0-a3c0-bce2b9050541\" (UID: \"78493013-3db7-46a0-a3c0-bce2b9050541\") " Oct 29 22:45:30.133102 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.133055 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume" pod "78493013-3db7-46a0-a3c0-bce2b9050541" (UID: "78493013-3db7-46a0-a3c0-bce2b9050541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:30.133226 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.133125 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume" pod "a2be3143-8369-43b8-83e3-99e21b36903d" (UID: "a2be3143-8369-43b8-83e3-99e21b36903d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:30.133466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.133347 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2be3143-8369-43b8-83e3-99e21b36903d-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.133466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.133373 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78493013-3db7-46a0-a3c0-bce2b9050541-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.134998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.134975 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj" pod "a2be3143-8369-43b8-83e3-99e21b36903d" (UID: "a2be3143-8369-43b8-83e3-99e21b36903d"). InnerVolumeSpecName "kube-api-access-gg9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:30.135127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.135008 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj" pod "78493013-3db7-46a0-a3c0-bce2b9050541" (UID: "78493013-3db7-46a0-a3c0-bce2b9050541"). InnerVolumeSpecName "kube-api-access-g6qvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:30.234213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.234191 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9rj\" (UniqueName: \"kubernetes.io/projected/a2be3143-8369-43b8-83e3-99e21b36903d-kube-api-access-gg9rj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.234213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.234213 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6qvj\" (UniqueName: \"kubernetes.io/projected/78493013-3db7-46a0-a3c0-bce2b9050541-kube-api-access-g6qvj\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:30.672984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.672892 4543 generic.go:356] "Generic (PLEG): container finished" podID="149ec293-c313-4360-b728-250da7da7388" containerID="97a9f264ce06d9fc76d2fb876e2821669d8fe258bdb5fe34f97ee6d17ce8f4f1" exitCode=0 Oct 29 22:45:30.673160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.672975 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" event={"ID":"149ec293-c313-4360-b728-250da7da7388","Type":"ContainerDied","Data":"97a9f264ce06d9fc76d2fb876e2821669d8fe258bdb5fe34f97ee6d17ce8f4f1"} Oct 29 22:45:30.674475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.674452 4543 generic.go:356] "Generic (PLEG): container finished" podID="78493013-3db7-46a0-a3c0-bce2b9050541" containerID="d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273" exitCode=0 Oct 29 22:45:30.674608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.674491 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" Oct 29 22:45:30.674608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.674531 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" event={"ID":"78493013-3db7-46a0-a3c0-bce2b9050541","Type":"ContainerDied","Data":"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273"} Oct 29 22:45:30.674608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.674576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d" event={"ID":"78493013-3db7-46a0-a3c0-bce2b9050541","Type":"ContainerDied","Data":"111fc8e43e4cf9c6ddd902c807ab62ce9d7dc233507b79bd943ed1a1b0786637"} Oct 29 22:45:30.674608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.674604 4543 scope.go:122] "RemoveContainer" containerID="d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273" Oct 29 22:45:30.676249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.676226 4543 generic.go:356] "Generic (PLEG): container finished" podID="a2be3143-8369-43b8-83e3-99e21b36903d" containerID="cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6" exitCode=0 Oct 29 22:45:30.676350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.676283 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" Oct 29 22:45:30.676350 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.676313 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" event={"ID":"a2be3143-8369-43b8-83e3-99e21b36903d","Type":"ContainerDied","Data":"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6"} Oct 29 22:45:30.676458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.676383 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz" event={"ID":"a2be3143-8369-43b8-83e3-99e21b36903d","Type":"ContainerDied","Data":"bebf37470a1164bd9bb1040ed89f16da24cfb93b6aa51b22b831ba806acb1999"} Oct 29 22:45:30.690216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.690188 4543 scope.go:122] "RemoveContainer" containerID="d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273" Oct 29 22:45:30.690540 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:30.690515 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273\": container with ID starting with d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273 not found: ID does not exist" containerID="d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273" Oct 29 22:45:30.690823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.690551 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273"} err="failed to get container status \"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273\": rpc error: code = NotFound desc = could not find container \"d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273\": container with ID starting with d6078f1d4a7107c0b8c7e11588a334e83dd3b23b77eff338cff0bf5dbb45e273 not found: ID does not exist" Oct 29 22:45:30.690823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.690575 4543 scope.go:122] "RemoveContainer" containerID="cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6" Oct 29 22:45:30.700359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.700343 4543 scope.go:122] "RemoveContainer" containerID="cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6" Oct 29 22:45:30.700625 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:30.700604 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6\": container with ID starting with cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6 not found: ID does not exist" containerID="cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6" Oct 29 22:45:30.700688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.700634 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6"} err="failed to get container status \"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6\": rpc error: code = NotFound desc = could not find container \"cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6\": container with ID starting with cbbb2e80a854bab26ed7d96eebc88e1fa54b63f888be07c7ff5c013f640b75e6 not found: ID does not exist" Oct 29 22:45:30.759312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.759287 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:45:30.767397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.767370 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4afaba-867bc4c4c8-4555d"] Oct 29 22:45:30.775196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.775175 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:45:30.775378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.775356 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" containerID="cri-o://46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b" gracePeriod=30 Oct 29 22:45:30.806890 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.806804 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:45:30.810435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:30.810413 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4afaba-cc54588bc-ppxpz"] Oct 29 22:45:31.829849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.829824 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:31.955467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.955402 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\") pod \"149ec293-c313-4360-b728-250da7da7388\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " Oct 29 22:45:31.955597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.955516 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\" (UniqueName: \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\") pod \"149ec293-c313-4360-b728-250da7da7388\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " Oct 29 22:45:31.955597 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.955555 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\") pod \"149ec293-c313-4360-b728-250da7da7388\" (UID: \"149ec293-c313-4360-b728-250da7da7388\") " Oct 29 22:45:31.955972 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.955940 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca" pod "149ec293-c313-4360-b728-250da7da7388" (UID: "149ec293-c313-4360-b728-250da7da7388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:31.957664 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.957641 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9" pod "149ec293-c313-4360-b728-250da7da7388" (UID: "149ec293-c313-4360-b728-250da7da7388"). InnerVolumeSpecName "kube-api-access-vpdr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:31.957760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:31.957731 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca" pod "149ec293-c313-4360-b728-250da7da7388" (UID: "149ec293-c313-4360-b728-250da7da7388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:32.056623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.056597 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdr9\" (UniqueName: \"kubernetes.io/projected/149ec293-c313-4360-b728-250da7da7388-kube-api-access-vpdr9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:32.056623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.056621 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149ec293-c313-4360-b728-250da7da7388-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:32.056750 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.056633 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149ec293-c313-4360-b728-250da7da7388-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:32.416144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.416105 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78493013-3db7-46a0-a3c0-bce2b9050541" path="/var/lib/kubelet/pods/78493013-3db7-46a0-a3c0-bce2b9050541/volumes" Oct 29 22:45:32.416702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.416687 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2be3143-8369-43b8-83e3-99e21b36903d" path="/var/lib/kubelet/pods/a2be3143-8369-43b8-83e3-99e21b36903d/volumes" Oct 29 22:45:32.694065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.693998 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" Oct 29 22:45:32.694065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.694025 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b713f27b-ptp8f" event={"ID":"149ec293-c313-4360-b728-250da7da7388","Type":"ContainerDied","Data":"bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51"} Oct 29 22:45:32.694267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:32.694068 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3afc63e35811bd8b4728dafe237ed699968a7ce3be249a18bd8a3ac0672e51" Oct 29 22:45:35.097465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:35.097427 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:35.097860 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:35.097491 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:36.052868 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.052817 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:45:36.099805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.099782 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.099849 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.099878 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.099943 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.099982 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100040 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100049 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100148 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100174 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100263 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:36.100626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100302 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\") pod \"8fb8c834-1ac4-41a2-a059-9df980779ae5\" (UID: \"8fb8c834-1ac4-41a2-a059-9df980779ae5\") " Oct 29 22:45:36.100626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100349 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:36.100626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100358 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:36.100885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.100862 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:36.101146 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.101127 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.101227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.101154 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.101227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.101171 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.101227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.101184 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.101227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.101197 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8c834-1ac4-41a2-a059-9df980779ae5-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.102179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.102159 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:36.102378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.102365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:36.102720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.102691 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:36.102783 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.102750 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s" pod "8fb8c834-1ac4-41a2-a059-9df980779ae5" (UID: "8fb8c834-1ac4-41a2-a059-9df980779ae5"). InnerVolumeSpecName "kube-api-access-spt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:36.202163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.202136 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.202163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.202157 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.202163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.202166 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8c834-1ac4-41a2-a059-9df980779ae5-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.202354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.202176 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spt7s\" (UniqueName: \"kubernetes.io/projected/8fb8c834-1ac4-41a2-a059-9df980779ae5-kube-api-access-spt7s\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:36.729168 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.729139 4543 generic.go:356] "Generic (PLEG): container finished" podID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerID="46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b" exitCode=0 Oct 29 22:45:36.729317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.729185 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" Oct 29 22:45:36.729317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.729223 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" event={"ID":"8fb8c834-1ac4-41a2-a059-9df980779ae5","Type":"ContainerDied","Data":"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b"} Oct 29 22:45:36.729317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.729274 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99" event={"ID":"8fb8c834-1ac4-41a2-a059-9df980779ae5","Type":"ContainerDied","Data":"3a6166b3287e73b14df4e3b792383508776c48e5ad2b3251232158a8fab71460"} Oct 29 22:45:36.729317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.729296 4543 scope.go:122] "RemoveContainer" containerID="46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b" Oct 29 22:45:36.740372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.740357 4543 scope.go:122] "RemoveContainer" containerID="46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b" Oct 29 22:45:36.740636 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:36.740614 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b\": container with ID starting with 46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b not found: ID does not exist" containerID="46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b" Oct 29 22:45:36.740707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.740648 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b"} err="failed to get container status \"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b\": rpc error: code = NotFound desc = could not find container \"46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b\": container with ID starting with 46759c68e61c8f155a3589b126af65209f4462c676b5f830f32ef822dc85298b not found: ID does not exist" Oct 29 22:45:36.804977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.804956 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:45:36.817841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:36.817811 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-pqh99"] Oct 29 22:45:38.385172 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:38.385141 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" path="/var/lib/kubelet/pods/8fb8c834-1ac4-41a2-a059-9df980779ae5/volumes" Oct 29 22:45:39.168816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:39.168780 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc"] Oct 29 22:45:39.181697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:39.181668 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09411-4t7hc"] Oct 29 22:45:40.262320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262287 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf"] Oct 29 22:45:40.262873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262859 4543 cpu_manager.go:414] "containerMap: removing container" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262876 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262887 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a2be3143-8369-43b8-83e3-99e21b36903d" containerName="payload-pre-processing" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262893 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2be3143-8369-43b8-83e3-99e21b36903d" containerName="payload-pre-processing" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262905 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262910 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:40.262923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262925 4543 cpu_manager.go:414] "containerMap: removing container" podUID="149ec293-c313-4360-b728-250da7da7388" containerName="revoke-keys" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262930 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149ec293-c313-4360-b728-250da7da7388" containerName="revoke-keys" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262938 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262943 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262966 4543 cpu_manager.go:414] "containerMap: removing container" podUID="78493013-3db7-46a0-a3c0-bce2b9050541" containerName="payload-processing" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.262970 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78493013-3db7-46a0-a3c0-bce2b9050541" containerName="payload-processing" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263057 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2be3143-8369-43b8-83e3-99e21b36903d" containerName="payload-pre-processing" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263066 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="98356f37-0e49-4513-99a2-d2f7c7f85ca7" containerName="cleanup" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263090 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="149ec293-c313-4360-b728-250da7da7388" containerName="revoke-keys" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263102 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb8c834-1ac4-41a2-a059-9df980779ae5" containerName="istio-proxy" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263113 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac5cbce-f0b2-4642-980f-c918af7c0747" containerName="cleanup" Oct 29 22:45:40.263159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.263125 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="78493013-3db7-46a0-a3c0-bce2b9050541" containerName="payload-processing" Oct 29 22:45:40.267945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.267927 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.285928 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.285906 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:40.289847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.289825 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf"] Oct 29 22:45:40.298281 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.298263 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:40.385615 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.385590 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d88eb8c-5b74-432b-9ecb-0d5c755a565b" path="/var/lib/kubelet/pods/0d88eb8c-5b74-432b-9ecb-0d5c755a565b/volumes" Oct 29 22:45:40.396492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.396472 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.396637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.396511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.396743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.396721 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8pzm\" (UniqueName: \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.497483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.497459 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.497594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.497503 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.497665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.497603 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8pzm\" (UniqueName: \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.498261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.498240 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.499976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.499958 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.532730 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.532680 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8pzm\" (UniqueName: \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\") pod \"maas-api-revoke-sub-b0fdac1e-p94bf\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.581652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.581631 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:40.887319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:40.887290 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf"] Oct 29 22:45:40.888772 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:40.888749 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20b8b47_7514_4bf4_84e2_deb5a2ddcc5e.slice/crio-8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429 WatchSource:0}: Error finding container 8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429: Status 404 returned error can't find the container with id 8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429 Oct 29 22:45:41.808163 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:41.808131 4543 generic.go:356] "Generic (PLEG): container finished" podID="a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" containerID="59cca94acce64576f983f0c49bb0c9298339496c171c4ae155e9b5cc29f435e9" exitCode=0 Oct 29 22:45:41.808533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:41.808206 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" event={"ID":"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e","Type":"ContainerDied","Data":"59cca94acce64576f983f0c49bb0c9298339496c171c4ae155e9b5cc29f435e9"} Oct 29 22:45:41.808533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:41.808257 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" event={"ID":"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e","Type":"ContainerStarted","Data":"8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429"} Oct 29 22:45:42.984295 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:42.984271 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:43.126473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.126372 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\") pod \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " Oct 29 22:45:43.126655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.126591 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\" (UniqueName: \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\") pod \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " Oct 29 22:45:43.126655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.126625 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\") pod \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\" (UID: \"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e\") " Oct 29 22:45:43.127244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.127217 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca" pod "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" (UID: "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:43.130305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.130273 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca" pod "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" (UID: "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:43.130414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.130392 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm" pod "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" (UID: "a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e"). InnerVolumeSpecName "kube-api-access-m8pzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:43.227949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.227925 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:43.227949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.227950 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:43.228114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.227961 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8pzm\" (UniqueName: \"kubernetes.io/projected/a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e-kube-api-access-m8pzm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:43.824282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.824255 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" Oct 29 22:45:43.824490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.824245 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0fdac1e-p94bf" event={"ID":"a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e","Type":"ContainerDied","Data":"8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429"} Oct 29 22:45:43.824490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:43.824390 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edda1ff8151da70d2df85f82434ce6924f5040e4d8fe80d20d8b29fd6633429" Oct 29 22:45:47.440811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.440773 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d"] Oct 29 22:45:47.441636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.441619 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" containerName="revoke-keys" Oct 29 22:45:47.441717 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.441639 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" containerName="revoke-keys" Oct 29 22:45:47.441808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.441795 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e" containerName="revoke-keys" Oct 29 22:45:47.445978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.445959 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.468677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.468654 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d"] Oct 29 22:45:47.554236 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.554203 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:47.554388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.554371 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:47.575602 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.575572 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.575702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.575636 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkt2x\" (UniqueName: \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.575743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.575727 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.677580 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.677543 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.677763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.677601 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkt2x\" (UniqueName: \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.677763 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.677642 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.678214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.678191 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.680120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.680095 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.704788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.704722 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkt2x\" (UniqueName: \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\") pod \"maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:47.760925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:47.760892 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:48.076492 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:48.076463 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d"] Oct 29 22:45:48.078911 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:48.078860 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbf8fad_7fd5_4f81_80a3_f1520f940ccb.slice/crio-7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da WatchSource:0}: Error finding container 7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da: Status 404 returned error can't find the container with id 7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da Oct 29 22:45:48.870207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:48.870169 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" event={"ID":"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb","Type":"ContainerStarted","Data":"5808c048e1ad512470343a914c506a63a5f2f89b429634e2aea7a12b61bcc2e0"} Oct 29 22:45:48.870795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:48.870230 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" event={"ID":"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb","Type":"ContainerStarted","Data":"7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da"} Oct 29 22:45:48.913732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:48.913669 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" podStartSLOduration=1.9136529009999999 podStartE2EDuration="1.913652901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:48.913004391 +0000 UTC m=+3810.654935063" watchObservedRunningTime="2026-10-29 22:45:48.913652901 +0000 UTC m=+3810.655583575" Oct 29 22:45:49.879062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:49.879028 4543 generic.go:356] "Generic (PLEG): container finished" podID="3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" containerID="5808c048e1ad512470343a914c506a63a5f2f89b429634e2aea7a12b61bcc2e0" exitCode=0 Oct 29 22:45:49.879491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:49.879113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" event={"ID":"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb","Type":"ContainerDied","Data":"5808c048e1ad512470343a914c506a63a5f2f89b429634e2aea7a12b61bcc2e0"} Oct 29 22:45:51.039605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.039577 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:51.120965 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.120928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\" (UniqueName: \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\") pod \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " Oct 29 22:45:51.121159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.121048 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\") pod \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " Oct 29 22:45:51.121285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.121264 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\") pod \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\" (UID: \"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb\") " Oct 29 22:45:51.121681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.121637 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca" pod "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" (UID: "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:51.121953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.121871 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:51.123138 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.123063 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x" pod "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" (UID: "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb"). InnerVolumeSpecName "kube-api-access-vkt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:51.123305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.123287 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca" pod "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" (UID: "3cbf8fad-7fd5-4f81-80a3-f1520f940ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:51.222971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.222946 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:51.222971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.222971 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkt2x\" (UniqueName: \"kubernetes.io/projected/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb-kube-api-access-vkt2x\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:51.894630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.894602 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" Oct 29 22:45:51.894818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.894639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d" event={"ID":"3cbf8fad-7fd5-4f81-80a3-f1520f940ccb","Type":"ContainerDied","Data":"7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da"} Oct 29 22:45:51.894818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:51.894688 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e5f0c5c4d5599806b1b2a754993ba78ca3ff3888404fcc0d03364c61ad263da" Oct 29 22:45:52.047462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.047429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv"] Oct 29 22:45:52.049895 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.049866 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq"] Oct 29 22:45:52.055160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.055130 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4b68d9-pzxfv"] Oct 29 22:45:52.058084 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.058052 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcebd115-rwzkq"] Oct 29 22:45:52.385988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.385955 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ff0d4d-ad15-4394-b254-8c1b22cf5b6c" path="/var/lib/kubelet/pods/67ff0d4d-ad15-4394-b254-8c1b22cf5b6c/volumes" Oct 29 22:45:52.386401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:52.386386 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96069476-078a-46bf-90e3-6e996d2fca77" path="/var/lib/kubelet/pods/96069476-078a-46bf-90e3-6e996d2fca77/volumes" Oct 29 22:45:54.613411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.613374 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6"] Oct 29 22:45:54.614273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.614249 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" containerName="revoke-keys" Oct 29 22:45:54.614273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.614273 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" containerName="revoke-keys" Oct 29 22:45:54.614462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.614441 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" containerName="revoke-keys" Oct 29 22:45:54.618407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.618386 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.637415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.637393 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6"] Oct 29 22:45:54.645408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.645389 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:54.645565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.645391 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:54.660522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.660477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2bz\" (UniqueName: \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.660900 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.660855 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.661061 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.660928 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.762431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.762397 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.762591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.762454 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.762591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.762582 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2bz\" (UniqueName: \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.763368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.763338 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.765044 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.765021 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.788820 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.788789 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2bz\" (UniqueName: \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\") pod \"maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:54.933688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:54.933607 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:55.250899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:55.250873 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6"] Oct 29 22:45:55.251988 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:45:55.251959 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c29a28_ab33_4822_8ac7_918b221c5e7b.slice/crio-6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7 WatchSource:0}: Error finding container 6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7: Status 404 returned error can't find the container with id 6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7 Oct 29 22:45:55.925517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:55.925425 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" event={"ID":"32c29a28-ab33-4822-8ac7-918b221c5e7b","Type":"ContainerStarted","Data":"0c6902b7f7dc1eae7c164600000c7fefc6d4d3971872a359e731e8615b2f02b1"} Oct 29 22:45:55.925517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:55.925484 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" event={"ID":"32c29a28-ab33-4822-8ac7-918b221c5e7b","Type":"ContainerStarted","Data":"6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7"} Oct 29 22:45:55.981243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:55.981193 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" podStartSLOduration=1.9811798349999998 podStartE2EDuration="1.981179835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:55.978994143 +0000 UTC m=+3817.720924816" watchObservedRunningTime="2026-10-29 22:45:55.981179835 +0000 UTC m=+3817.723110508" Oct 29 22:45:56.934608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:56.934574 4543 generic.go:356] "Generic (PLEG): container finished" podID="32c29a28-ab33-4822-8ac7-918b221c5e7b" containerID="0c6902b7f7dc1eae7c164600000c7fefc6d4d3971872a359e731e8615b2f02b1" exitCode=0 Oct 29 22:45:56.935019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:56.934667 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" event={"ID":"32c29a28-ab33-4822-8ac7-918b221c5e7b","Type":"ContainerDied","Data":"0c6902b7f7dc1eae7c164600000c7fefc6d4d3971872a359e731e8615b2f02b1"} Oct 29 22:45:57.913644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:57.913613 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:45:57.969336 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:45:57.969302 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:45:58.134059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.134016 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:58.213829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.213789 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\") pod \"32c29a28-ab33-4822-8ac7-918b221c5e7b\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " Oct 29 22:45:58.214023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.213929 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\" (UniqueName: \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\") pod \"32c29a28-ab33-4822-8ac7-918b221c5e7b\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " Oct 29 22:45:58.214150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.214118 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\") pod \"32c29a28-ab33-4822-8ac7-918b221c5e7b\" (UID: \"32c29a28-ab33-4822-8ac7-918b221c5e7b\") " Oct 29 22:45:58.214206 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.214186 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca" pod "32c29a28-ab33-4822-8ac7-918b221c5e7b" (UID: "32c29a28-ab33-4822-8ac7-918b221c5e7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:58.214825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.214755 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:58.216384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.216363 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca" pod "32c29a28-ab33-4822-8ac7-918b221c5e7b" (UID: "32c29a28-ab33-4822-8ac7-918b221c5e7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:58.216485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.216383 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz" pod "32c29a28-ab33-4822-8ac7-918b221c5e7b" (UID: "32c29a28-ab33-4822-8ac7-918b221c5e7b"). InnerVolumeSpecName "kube-api-access-vn2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:58.315940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.315899 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn2bz\" (UniqueName: \"kubernetes.io/projected/32c29a28-ab33-4822-8ac7-918b221c5e7b-kube-api-access-vn2bz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:58.315940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.315939 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c29a28-ab33-4822-8ac7-918b221c5e7b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:45:58.954099 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.954046 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" Oct 29 22:45:58.954277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.954100 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6" event={"ID":"32c29a28-ab33-4822-8ac7-918b221c5e7b","Type":"ContainerDied","Data":"6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7"} Oct 29 22:45:58.954277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:58.954148 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ada764b5f1875d471c1a94ec854cf7bf80704f0be5445f066c0c3485f82e0f7" Oct 29 22:45:59.641991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:59.641923 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:45:59.677120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:45:59.677091 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:46:02.914244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:02.914203 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" podUID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" containerName="payload-processing" containerID="cri-o://a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd" gracePeriod=25 Oct 29 22:46:03.179424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.179368 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:46:03.232789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.232762 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\") pod \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " Oct 29 22:46:03.232954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.232871 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\" (UniqueName: \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\") pod \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\" (UID: \"b213f6c4-c1cf-4232-8b83-4d905a24e8f1\") " Oct 29 22:46:03.233159 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.233132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume" pod "b213f6c4-c1cf-4232-8b83-4d905a24e8f1" (UID: "b213f6c4-c1cf-4232-8b83-4d905a24e8f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:03.235003 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.234980 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz" pod "b213f6c4-c1cf-4232-8b83-4d905a24e8f1" (UID: "b213f6c4-c1cf-4232-8b83-4d905a24e8f1"). InnerVolumeSpecName "kube-api-access-nnpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:03.333811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.333784 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:03.333811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.333810 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnpcz\" (UniqueName: \"kubernetes.io/projected/b213f6c4-c1cf-4232-8b83-4d905a24e8f1-kube-api-access-nnpcz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:03.998661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.998630 4543 generic.go:356] "Generic (PLEG): container finished" podID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" containerID="a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd" exitCode=0 Oct 29 22:46:03.999095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.998681 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" Oct 29 22:46:03.999095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.998707 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" event={"ID":"b213f6c4-c1cf-4232-8b83-4d905a24e8f1","Type":"ContainerDied","Data":"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd"} Oct 29 22:46:03.999095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.998770 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl" event={"ID":"b213f6c4-c1cf-4232-8b83-4d905a24e8f1","Type":"ContainerDied","Data":"892a6642a1186c4d69554a400ce49a21b97b2758825af31a69ebf24279d111ff"} Oct 29 22:46:03.999095 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:03.998799 4543 scope.go:122] "RemoveContainer" containerID="a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd" Oct 29 22:46:04.011066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.011050 4543 scope.go:122] "RemoveContainer" containerID="a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd" Oct 29 22:46:04.011371 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:04.011350 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd\": container with ID starting with a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd not found: ID does not exist" containerID="a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd" Oct 29 22:46:04.011424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.011378 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd"} err="failed to get container status \"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd\": rpc error: code = NotFound desc = could not find container \"a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd\": container with ID starting with a5a78f84ed7001ca94c286b967471617c1cd741e30756421f468a12529badabd not found: ID does not exist" Oct 29 22:46:04.056966 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.056911 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:46:04.070227 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.070209 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-563f39-5f566cc9fd-lpgkl"] Oct 29 22:46:04.385920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.385863 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" path="/var/lib/kubelet/pods/b213f6c4-c1cf-4232-8b83-4d905a24e8f1/volumes" Oct 29 22:46:04.642798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.642717 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" podUID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" containerName="payload-processing" containerID="cri-o://bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15" gracePeriod=25 Oct 29 22:46:04.678011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.677981 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" podUID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" containerName="payload-pre-processing" containerID="cri-o://da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b" gracePeriod=25 Oct 29 22:46:04.935508 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.935481 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:46:04.948872 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.948849 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\") pod \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " Oct 29 22:46:04.949022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.948949 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\" (UniqueName: \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\") pod \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\" (UID: \"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca\") " Oct 29 22:46:04.949307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.949280 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume" pod "c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" (UID: "c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:04.949423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.949404 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:04.951325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.951305 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk" pod "c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" (UID: "c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca"). InnerVolumeSpecName "kube-api-access-lhlrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:04.968709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:04.968689 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:46:05.009531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.009503 4543 generic.go:356] "Generic (PLEG): container finished" podID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" containerID="da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b" exitCode=0 Oct 29 22:46:05.009968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.009554 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" Oct 29 22:46:05.009968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.009592 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" event={"ID":"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46","Type":"ContainerDied","Data":"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b"} Oct 29 22:46:05.009968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.009638 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54" event={"ID":"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46","Type":"ContainerDied","Data":"193c838df434f11d5b0485da4525cdfeb292becd9ed64389d1866ac8ecb6077b"} Oct 29 22:46:05.009968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.009665 4543 scope.go:122] "RemoveContainer" containerID="da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b" Oct 29 22:46:05.011323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.011301 4543 generic.go:356] "Generic (PLEG): container finished" podID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" containerID="bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15" exitCode=0 Oct 29 22:46:05.011451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.011333 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" Oct 29 22:46:05.011451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.011346 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" event={"ID":"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca","Type":"ContainerDied","Data":"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15"} Oct 29 22:46:05.011451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.011385 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524" event={"ID":"c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca","Type":"ContainerDied","Data":"d349865dbc8e71ed21e6fff07ef488308acf26d311a614dc8601322007887d34"} Oct 29 22:46:05.021115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.021096 4543 scope.go:122] "RemoveContainer" containerID="da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b" Oct 29 22:46:05.021423 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:05.021400 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b\": container with ID starting with da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b not found: ID does not exist" containerID="da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b" Oct 29 22:46:05.021484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.021434 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b"} err="failed to get container status \"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b\": rpc error: code = NotFound desc = could not find container \"da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b\": container with ID starting with da30c31d17885553ddd05b5089b90c434521291db4b7d584d8566a0263bcfa0b not found: ID does not exist" Oct 29 22:46:05.021484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.021458 4543 scope.go:122] "RemoveContainer" containerID="bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15" Oct 29 22:46:05.035052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.035028 4543 scope.go:122] "RemoveContainer" containerID="bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15" Oct 29 22:46:05.035417 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:05.035397 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15\": container with ID starting with bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15 not found: ID does not exist" containerID="bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15" Oct 29 22:46:05.035517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.035423 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15"} err="failed to get container status \"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15\": rpc error: code = NotFound desc = could not find container \"bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15\": container with ID starting with bfcf3e99881615165d7a932b609989971c22f5c25b17286b3036c2ce9c5a5d15 not found: ID does not exist" Oct 29 22:46:05.050489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.050468 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlrk\" (UniqueName: \"kubernetes.io/projected/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca-kube-api-access-lhlrk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:05.085527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.085502 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:46:05.088708 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.088687 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:46:05.091675 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.091642 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f52917-657fbdcb7d-6m524"] Oct 29 22:46:05.151140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.151112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\" (UniqueName: \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\") pod \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " Oct 29 22:46:05.151239 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.151162 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\") pod \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\" (UID: \"d7a0e0fb-64ad-4de5-8b91-c03f74d78f46\") " Oct 29 22:46:05.151517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.151496 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume" pod "d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" (UID: "d7a0e0fb-64ad-4de5-8b91-c03f74d78f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:05.153474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.153450 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd" pod "d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" (UID: "d7a0e0fb-64ad-4de5-8b91-c03f74d78f46"). InnerVolumeSpecName "kube-api-access-nh2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:05.252584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.252549 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2sd\" (UniqueName: \"kubernetes.io/projected/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-kube-api-access-nh2sd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:05.252746 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.252590 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:05.387846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.387812 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:46:05.395490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:05.395466 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f52917-679fb8bc6c-sgb54"] Oct 29 22:46:06.385453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:06.385418 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" path="/var/lib/kubelet/pods/c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca/volumes" Oct 29 22:46:06.385840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:06.385817 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" path="/var/lib/kubelet/pods/d7a0e0fb-64ad-4de5-8b91-c03f74d78f46/volumes" Oct 29 22:46:10.089393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:10.089357 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" containerID="cri-o://0cfb1ec39ade7c110fde493ca4a2c4790e10fe0308635b7d13261d0643f8565d" gracePeriod=25 Oct 29 22:46:10.115114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:10.115063 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:11.883736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:11.883701 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d"] Oct 29 22:46:11.896480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:11.896454 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-563f39-cfc4d"] Oct 29 22:46:12.384835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:12.384800 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cbf8fad-7fd5-4f81-80a3-f1520f940ccb" path="/var/lib/kubelet/pods/3cbf8fad-7fd5-4f81-80a3-f1520f940ccb/volumes" Oct 29 22:46:16.408144 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:16.408093 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:46:16.408649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:16.408355 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" containerID="cri-o://abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367" gracePeriod=30 Oct 29 22:46:16.490632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:16.490596 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:46:16.490885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:16.490837 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" podUID="bf56622f-3488-40de-8962-13187f47e0d4" containerName="maas-api" containerID="cri-o://bc9bcf54ba0575775d86902782d159d44cdef1ac9b204557a6aa6a0e58a88615" gracePeriod=30 Oct 29 22:46:17.168123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.168013 4543 generic.go:356] "Generic (PLEG): container finished" podID="bf56622f-3488-40de-8962-13187f47e0d4" containerID="bc9bcf54ba0575775d86902782d159d44cdef1ac9b204557a6aa6a0e58a88615" exitCode=0 Oct 29 22:46:17.168123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.168107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" event={"ID":"bf56622f-3488-40de-8962-13187f47e0d4","Type":"ContainerDied","Data":"bc9bcf54ba0575775d86902782d159d44cdef1ac9b204557a6aa6a0e58a88615"} Oct 29 22:46:17.297692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.297653 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:17.297853 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.297709 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:17.304427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.304399 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6"] Oct 29 22:46:17.317541 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.317517 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27b6db-kwxz6"] Oct 29 22:46:17.409225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409194 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd"] Oct 29 22:46:17.409873 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409855 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" containerName="payload-pre-processing" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409875 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" containerName="payload-pre-processing" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409896 4543 cpu_manager.go:414] "containerMap: removing container" podUID="32c29a28-ab33-4822-8ac7-918b221c5e7b" containerName="revoke-keys" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409902 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c29a28-ab33-4822-8ac7-918b221c5e7b" containerName="revoke-keys" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409918 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" containerName="payload-processing" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409923 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" containerName="payload-processing" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409936 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" containerName="payload-processing" Oct 29 22:46:17.409942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.409941 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" containerName="payload-processing" Oct 29 22:46:17.410341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.410032 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5035aff-e5d5-4720-b9ad-5bb32b2cc3ca" containerName="payload-processing" Oct 29 22:46:17.410341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.410050 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a0e0fb-64ad-4de5-8b91-c03f74d78f46" containerName="payload-pre-processing" Oct 29 22:46:17.410341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.410058 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c29a28-ab33-4822-8ac7-918b221c5e7b" containerName="revoke-keys" Oct 29 22:46:17.410341 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.410066 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b213f6c4-c1cf-4232-8b83-4d905a24e8f1" containerName="payload-processing" Oct 29 22:46:17.413563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.413546 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.437777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.437753 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd"] Oct 29 22:46:17.438403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.438377 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:46:17.438538 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.438487 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:46:17.446094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.444562 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5dp2\" (UniqueName: \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.446094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.444601 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.446094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.445542 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.493474 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.493455 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:46:17.547762 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.547671 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\") pod \"bf56622f-3488-40de-8962-13187f47e0d4\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " Oct 29 22:46:17.547907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.547773 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\" (UniqueName: \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\") pod \"bf56622f-3488-40de-8962-13187f47e0d4\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " Oct 29 22:46:17.547907 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.547834 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\") pod \"bf56622f-3488-40de-8962-13187f47e0d4\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " Oct 29 22:46:17.548037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.547954 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\") pod \"bf56622f-3488-40de-8962-13187f47e0d4\" (UID: \"bf56622f-3488-40de-8962-13187f47e0d4\") " Oct 29 22:46:17.548225 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.548199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.550529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.548399 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5dp2\" (UniqueName: \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.550529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.548431 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.550529 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.548513 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca" pod "bf56622f-3488-40de-8962-13187f47e0d4" (UID: "bf56622f-3488-40de-8962-13187f47e0d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:17.550767 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.550631 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls" pod "bf56622f-3488-40de-8962-13187f47e0d4" (UID: "bf56622f-3488-40de-8962-13187f47e0d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:17.551428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.550994 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs" pod "bf56622f-3488-40de-8962-13187f47e0d4" (UID: "bf56622f-3488-40de-8962-13187f47e0d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:17.551524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.551463 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z" pod "bf56622f-3488-40de-8962-13187f47e0d4" (UID: "bf56622f-3488-40de-8962-13187f47e0d4"). InnerVolumeSpecName "kube-api-access-z845z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:17.551770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.551745 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.553519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.553497 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.577216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.577186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5dp2\" (UniqueName: \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:17.650290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.650260 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-metrics-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:17.650290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.650290 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf56622f-3488-40de-8962-13187f47e0d4-maas-api-tls\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:17.650522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.650303 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z845z\" (UniqueName: \"kubernetes.io/projected/bf56622f-3488-40de-8962-13187f47e0d4-kube-api-access-z845z\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:17.650522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.650316 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf56622f-3488-40de-8962-13187f47e0d4-openshift-service-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:17.731514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:17.731486 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:18.042184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.042159 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd"] Oct 29 22:46:18.043604 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:18.043570 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeff3914_9805_4645_831c_a7fd796a4787.slice/crio-096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627 WatchSource:0}: Error finding container 096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627: Status 404 returned error can't find the container with id 096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627 Oct 29 22:46:18.177240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.177211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" event={"ID":"aeff3914-9805-4645-831c-a7fd796a4787","Type":"ContainerStarted","Data":"096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627"} Oct 29 22:46:18.178600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.178568 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" event={"ID":"bf56622f-3488-40de-8962-13187f47e0d4","Type":"ContainerDied","Data":"a7bc7dc221621dccd1deab965a6659734bcb505c59c580cc943a4e7b13051627"} Oct 29 22:46:18.178719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.178623 4543 scope.go:122] "RemoveContainer" containerID="bc9bcf54ba0575775d86902782d159d44cdef1ac9b204557a6aa6a0e58a88615" Oct 29 22:46:18.178719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.178577 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc" Oct 29 22:46:18.241124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.241103 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:46:18.248241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.248219 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27b6db-746db8b8cf-b5wcc"] Oct 29 22:46:18.385329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.385299 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c29a28-ab33-4822-8ac7-918b221c5e7b" path="/var/lib/kubelet/pods/32c29a28-ab33-4822-8ac7-918b221c5e7b/volumes" Oct 29 22:46:18.385792 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:18.385776 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf56622f-3488-40de-8962-13187f47e0d4" path="/var/lib/kubelet/pods/bf56622f-3488-40de-8962-13187f47e0d4/volumes" Oct 29 22:46:19.196513 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:19.196427 4543 generic.go:356] "Generic (PLEG): container finished" podID="aeff3914-9805-4645-831c-a7fd796a4787" containerID="5534f4b6c41c0a8f8123b58a96cca7302c0dc86d7118d015c92f01d471ce7906" exitCode=0 Oct 29 22:46:19.196894 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:19.196506 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" event={"ID":"aeff3914-9805-4645-831c-a7fd796a4787","Type":"ContainerDied","Data":"5534f4b6c41c0a8f8123b58a96cca7302c0dc86d7118d015c92f01d471ce7906"} Oct 29 22:46:20.115255 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.115217 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:20.133937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.133906 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:46:20.134199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.134118 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" containerID="cri-o://4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085" gracePeriod=30 Oct 29 22:46:20.406193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.406166 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:20.464448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.464419 4543 patch_prober.go:29] interesting pod/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:20.464600 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.464486 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:20.489415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.489392 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\" (UniqueName: \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\") pod \"aeff3914-9805-4645-831c-a7fd796a4787\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " Oct 29 22:46:20.489561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.489526 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\") pod \"aeff3914-9805-4645-831c-a7fd796a4787\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " Oct 29 22:46:20.489639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.489606 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\") pod \"aeff3914-9805-4645-831c-a7fd796a4787\" (UID: \"aeff3914-9805-4645-831c-a7fd796a4787\") " Oct 29 22:46:20.489957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.489925 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca" pod "aeff3914-9805-4645-831c-a7fd796a4787" (UID: "aeff3914-9805-4645-831c-a7fd796a4787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:20.491653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.491624 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2" pod "aeff3914-9805-4645-831c-a7fd796a4787" (UID: "aeff3914-9805-4645-831c-a7fd796a4787"). InnerVolumeSpecName "kube-api-access-f5dp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:20.491805 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.491786 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca" pod "aeff3914-9805-4645-831c-a7fd796a4787" (UID: "aeff3914-9805-4645-831c-a7fd796a4787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:20.591475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.591449 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:20.591601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.591478 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeff3914-9805-4645-831c-a7fd796a4787-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:20.591601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.591488 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5dp2\" (UniqueName: \"kubernetes.io/projected/aeff3914-9805-4645-831c-a7fd796a4787-kube-api-access-f5dp2\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:20.719145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719114 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq"] Oct 29 22:46:20.719760 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719741 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bf56622f-3488-40de-8962-13187f47e0d4" containerName="maas-api" Oct 29 22:46:20.719899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719762 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf56622f-3488-40de-8962-13187f47e0d4" containerName="maas-api" Oct 29 22:46:20.719899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719781 4543 cpu_manager.go:414] "containerMap: removing container" podUID="aeff3914-9805-4645-831c-a7fd796a4787" containerName="revoke-keys" Oct 29 22:46:20.719899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719788 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeff3914-9805-4645-831c-a7fd796a4787" containerName="revoke-keys" Oct 29 22:46:20.720094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719982 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf56622f-3488-40de-8962-13187f47e0d4" containerName="maas-api" Oct 29 22:46:20.720094 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.719999 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeff3914-9805-4645-831c-a7fd796a4787" containerName="revoke-keys" Oct 29 22:46:20.723701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.723681 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.759831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.759806 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq"] Oct 29 22:46:20.799460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.799433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.799601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.799509 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.799601 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.799570 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.900488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.900463 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.900604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.900529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.900604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.900565 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.901244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.901222 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.902937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.902919 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:20.926194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:20.926169 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:21.037387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.037337 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:21.213647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.213623 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" Oct 29 22:46:21.213788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.213661 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd" event={"ID":"aeff3914-9805-4645-831c-a7fd796a4787","Type":"ContainerDied","Data":"096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627"} Oct 29 22:46:21.213788 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.213708 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096ecc7ff68888ce97fccb7bbf70e1f3fb715f2485cffdfa8237439c21a6e627" Oct 29 22:46:21.366058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.366025 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq"] Oct 29 22:46:21.368475 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:21.368443 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f58773_a35e_4d40_9316_fcbc178ed884.slice/crio-4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf WatchSource:0}: Error finding container 4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf: Status 404 returned error can't find the container with id 4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf Oct 29 22:46:21.747937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.747914 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:46:21.811967 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.811941 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812006 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812053 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812149 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812200 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812267 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812252 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812297 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812349 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:21.812445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812385 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\") pod \"76ad355e-7236-4438-9ade-e4978b91f0c0\" (UID: \"76ad355e-7236-4438-9ade-e4978b91f0c0\") " Oct 29 22:46:21.812563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812454 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:21.812563 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812461 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:21.812697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812572 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:21.812803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.812780 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:21.813058 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.813023 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.813196 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.813107 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.813324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.813260 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.813324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.813277 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ad355e-7236-4438-9ade-e4978b91f0c0-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.813324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.813290 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.814390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.814364 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "kube-api-access-n6xjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:21.814503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.814481 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:21.814579 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.814565 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:21.814660 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.814643 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo" pod "76ad355e-7236-4438-9ade-e4978b91f0c0" (UID: "76ad355e-7236-4438-9ade-e4978b91f0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:21.914331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.914266 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xjh\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-kube-api-access-n6xjh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.914331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.914288 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.914331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.914298 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:21.914331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:21.914306 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ad355e-7236-4438-9ade-e4978b91f0c0-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:22.221901 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.221867 4543 generic.go:356] "Generic (PLEG): container finished" podID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerID="abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367" exitCode=0 Oct 29 22:46:22.222123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.221926 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" Oct 29 22:46:22.222123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.221962 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" event={"ID":"76ad355e-7236-4438-9ade-e4978b91f0c0","Type":"ContainerDied","Data":"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367"} Oct 29 22:46:22.222123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.222019 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9" event={"ID":"76ad355e-7236-4438-9ade-e4978b91f0c0","Type":"ContainerDied","Data":"980bfbc61c1fa240b1a41ff352b9a5c207b337fea1647a92fdf0cd3a2e5e36b0"} Oct 29 22:46:22.222123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.222046 4543 scope.go:122] "RemoveContainer" containerID="abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367" Oct 29 22:46:22.223959 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.223937 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" event={"ID":"22f58773-a35e-4d40-9316-fcbc178ed884","Type":"ContainerStarted","Data":"75d9ba155a11156a6a9c19afa39b670a10078ddcc809ddf32b7a5935cde4679e"} Oct 29 22:46:22.224100 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.223975 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" event={"ID":"22f58773-a35e-4d40-9316-fcbc178ed884","Type":"ContainerStarted","Data":"4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf"} Oct 29 22:46:22.234412 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.234396 4543 scope.go:122] "RemoveContainer" containerID="abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367" Oct 29 22:46:22.234671 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:22.234652 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367\": container with ID starting with abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367 not found: ID does not exist" containerID="abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367" Oct 29 22:46:22.234740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.234682 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367"} err="failed to get container status \"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367\": rpc error: code = NotFound desc = could not find container \"abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367\": container with ID starting with abc9a81faf689434ea87bad7aa586f9d27680e9c77acedb187969e60f2986367 not found: ID does not exist" Oct 29 22:46:22.285849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.285811 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" podStartSLOduration=2.285799297 podStartE2EDuration="2.285799297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:22.28300645 +0000 UTC m=+3844.024937134" watchObservedRunningTime="2026-10-29 22:46:22.285799297 +0000 UTC m=+3844.027730006" Oct 29 22:46:22.326040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.326018 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:46:22.335656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.335627 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-8lgp9"] Oct 29 22:46:22.386356 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:22.386191 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" path="/var/lib/kubelet/pods/76ad355e-7236-4438-9ade-e4978b91f0c0/volumes" Oct 29 22:46:23.239135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:23.238998 4543 generic.go:356] "Generic (PLEG): container finished" podID="22f58773-a35e-4d40-9316-fcbc178ed884" containerID="75d9ba155a11156a6a9c19afa39b670a10078ddcc809ddf32b7a5935cde4679e" exitCode=0 Oct 29 22:46:23.239135 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:23.239120 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" event={"ID":"22f58773-a35e-4d40-9316-fcbc178ed884","Type":"ContainerDied","Data":"75d9ba155a11156a6a9c19afa39b670a10078ddcc809ddf32b7a5935cde4679e"} Oct 29 22:46:24.395519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.395499 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:24.545945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.545883 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\") pod \"22f58773-a35e-4d40-9316-fcbc178ed884\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " Oct 29 22:46:24.545945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.545921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\") pod \"22f58773-a35e-4d40-9316-fcbc178ed884\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " Oct 29 22:46:24.546202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.545957 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\") pod \"22f58773-a35e-4d40-9316-fcbc178ed884\" (UID: \"22f58773-a35e-4d40-9316-fcbc178ed884\") " Oct 29 22:46:24.546333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.546311 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca" pod "22f58773-a35e-4d40-9316-fcbc178ed884" (UID: "22f58773-a35e-4d40-9316-fcbc178ed884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:24.548304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.548279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75" pod "22f58773-a35e-4d40-9316-fcbc178ed884" (UID: "22f58773-a35e-4d40-9316-fcbc178ed884"). InnerVolumeSpecName "kube-api-access-pwx75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:24.548405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.548321 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca" pod "22f58773-a35e-4d40-9316-fcbc178ed884" (UID: "22f58773-a35e-4d40-9316-fcbc178ed884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:24.647393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.647373 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:24.647393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.647394 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/22f58773-a35e-4d40-9316-fcbc178ed884-kube-api-access-pwx75\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:24.647523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:24.647404 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f58773-a35e-4d40-9316-fcbc178ed884-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.182779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.182743 4543 scope.go:122] "RemoveContainer" containerID="41961e1cd74505da4b412ff2e2dc3e98b830f644e14cdaa568aa5e68ce30f143" Oct 29 22:46:25.258642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.258192 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" Oct 29 22:46:25.258642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.258253 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq" event={"ID":"22f58773-a35e-4d40-9316-fcbc178ed884","Type":"ContainerDied","Data":"4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf"} Oct 29 22:46:25.258642 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.258319 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4446733f91487430e94465581b145a41ffb085ba68689fb14b403ae8bf846acf" Oct 29 22:46:25.259155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.259136 4543 scope.go:122] "RemoveContainer" containerID="a7605fea59ac0270b2f5da948f8ec90e1967141a682eef97d0e085e2b42edaa2" Oct 29 22:46:25.269069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.269053 4543 scope.go:122] "RemoveContainer" containerID="fc254f1f7c20fdc09517965977411de456c305e7320686a343309d3b5f439ada" Oct 29 22:46:25.279463 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.279442 4543 scope.go:122] "RemoveContainer" containerID="67e07e266c30aaef4989066ff9737869385502be9717c212637dca84ba121609" Oct 29 22:46:25.288702 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.288681 4543 scope.go:122] "RemoveContainer" containerID="41a095872413f1856c748c05e4c0c14757ab397a2e674fd111eb045bf95ee8a4" Oct 29 22:46:25.300676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.300656 4543 scope.go:122] "RemoveContainer" containerID="8d01908378aeea88ffb8b6cbb84a2c7090ca68f234b7f9cbb4707f3c945c088e" Oct 29 22:46:25.430659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.430638 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:46:25.558387 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558360 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558554 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558491 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558620 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558603 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558637 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558654 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.558778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558678 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558721 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558755 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558764 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.558935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558809 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.558935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.558841 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\") pod \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\" (UID: \"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303\") " Oct 29 22:46:25.559256 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559228 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.559366 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:25.559484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559460 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.559559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.559559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559493 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.559559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559508 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.559559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.559523 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.561272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.561249 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:25.561398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.561304 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.562012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.561964 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:25.562012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.561969 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw" pod "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" (UID: "d4bbf528-f2f4-4ca7-8e9f-f09af5d74303"). InnerVolumeSpecName "kube-api-access-m8nkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.661193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.661140 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8nkw\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-kube-api-access-m8nkw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.661193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.661164 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.661193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.661175 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.661193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.661183 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.661193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:25.661192 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.110945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.110916 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g"] Oct 29 22:46:26.111480 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111467 4543 cpu_manager.go:414] "containerMap: removing container" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" Oct 29 22:46:26.111552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111482 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" Oct 29 22:46:26.111552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111496 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:26.111552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111512 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:26.111552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111543 4543 cpu_manager.go:414] "containerMap: removing container" podUID="22f58773-a35e-4d40-9316-fcbc178ed884" containerName="revoke-keys" Oct 29 22:46:26.111552 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111548 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f58773-a35e-4d40-9316-fcbc178ed884" containerName="revoke-keys" Oct 29 22:46:26.111719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111638 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f58773-a35e-4d40-9316-fcbc178ed884" containerName="revoke-keys" Oct 29 22:46:26.111719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111646 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:26.111719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111657 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ad355e-7236-4438-9ade-e4978b91f0c0" containerName="istio-proxy" Oct 29 22:46:26.111719 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.111664 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:26.116266 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.116250 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.142305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.142281 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g"] Oct 29 22:46:26.151026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.150988 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:46:26.151026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.150999 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:46:26.204542 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.204516 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:46:26.204669 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.204550 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:46:26.265968 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.265941 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.266115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.265996 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.266115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.266103 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nrx\" (UniqueName: \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.267430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.267404 4543 generic.go:356] "Generic (PLEG): container finished" podID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerID="4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085" exitCode=0 Oct 29 22:46:26.267547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.267453 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" Oct 29 22:46:26.267547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.267497 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerDied","Data":"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085"} Oct 29 22:46:26.267676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.267556 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q" event={"ID":"d4bbf528-f2f4-4ca7-8e9f-f09af5d74303","Type":"ContainerDied","Data":"b1fb834f4d2bc2df903f72b564830206191593b62abf7ef37ad5e73e9a7c8647"} Oct 29 22:46:26.267676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.267603 4543 scope.go:122] "RemoveContainer" containerID="4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085" Oct 29 22:46:26.279184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.279169 4543 scope.go:122] "RemoveContainer" containerID="fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83" Oct 29 22:46:26.288933 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.288911 4543 scope.go:122] "RemoveContainer" containerID="4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085" Oct 29 22:46:26.289232 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:26.289213 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085\": container with ID starting with 4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085 not found: ID does not exist" containerID="4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085" Oct 29 22:46:26.289323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.289240 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085"} err="failed to get container status \"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085\": rpc error: code = NotFound desc = could not find container \"4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085\": container with ID starting with 4bb53dbae215917481961d21357ec369dd50cf7e74c1df35d9b0a373b49ca085 not found: ID does not exist" Oct 29 22:46:26.289323 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.289257 4543 scope.go:122] "RemoveContainer" containerID="fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83" Oct 29 22:46:26.289531 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:26.289511 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83\": container with ID starting with fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83 not found: ID does not exist" containerID="fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83" Oct 29 22:46:26.289609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.289536 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83"} err="failed to get container status \"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83\": rpc error: code = NotFound desc = could not find container \"fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83\": container with ID starting with fc539372a6f0adae6006249a171e3d7e007af9fab8e3808b7d245c2818ab6c83 not found: ID does not exist" Oct 29 22:46:26.334688 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.334664 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:46:26.347922 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.347902 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-563f39-openshift-default-6686544bbf-wxn6q"] Oct 29 22:46:26.367489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.367443 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.367489 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.367479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.367641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.367513 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nrx\" (UniqueName: \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.368148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.368129 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.369904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.369887 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.385208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.385185 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" path="/var/lib/kubelet/pods/d4bbf528-f2f4-4ca7-8e9f-f09af5d74303/volumes" Oct 29 22:46:26.405698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.405678 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nrx\" (UniqueName: \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\") pod \"maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.429940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.429911 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:26.736419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:26.736390 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g"] Oct 29 22:46:26.738042 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:26.738017 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbace6f6_99d5_4322_a55f_ec8eba406b02.slice/crio-0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462 WatchSource:0}: Error finding container 0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462: Status 404 returned error can't find the container with id 0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462 Oct 29 22:46:27.276388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:27.276354 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" event={"ID":"cbace6f6-99d5-4322-a55f-ec8eba406b02","Type":"ContainerStarted","Data":"9bfd288c7f7ca884860b7aad9ef07e2b691dce99dc68023e4097975ec6978301"} Oct 29 22:46:27.276528 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:27.276449 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" event={"ID":"cbace6f6-99d5-4322-a55f-ec8eba406b02","Type":"ContainerStarted","Data":"0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462"} Oct 29 22:46:27.321482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:27.321405 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" podStartSLOduration=1.3213927970000001 podStartE2EDuration="1.321392797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:27.319838632 +0000 UTC m=+3849.061769300" watchObservedRunningTime="2026-10-29 22:46:27.321392797 +0000 UTC m=+3849.063323470" Oct 29 22:46:27.960683 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:27.960599 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:46:27.977866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:27.977842 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 22:46:28.285951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:28.285915 4543 generic.go:356] "Generic (PLEG): container finished" podID="cbace6f6-99d5-4322-a55f-ec8eba406b02" containerID="9bfd288c7f7ca884860b7aad9ef07e2b691dce99dc68023e4097975ec6978301" exitCode=0 Oct 29 22:46:28.286186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:28.286000 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" event={"ID":"cbace6f6-99d5-4322-a55f-ec8eba406b02","Type":"ContainerDied","Data":"9bfd288c7f7ca884860b7aad9ef07e2b691dce99dc68023e4097975ec6978301"} Oct 29 22:46:29.453881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.453854 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:29.614445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.614382 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\") pod \"cbace6f6-99d5-4322-a55f-ec8eba406b02\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " Oct 29 22:46:29.614582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.614445 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\" (UniqueName: \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\") pod \"cbace6f6-99d5-4322-a55f-ec8eba406b02\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " Oct 29 22:46:29.614582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.614567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\") pod \"cbace6f6-99d5-4322-a55f-ec8eba406b02\" (UID: \"cbace6f6-99d5-4322-a55f-ec8eba406b02\") " Oct 29 22:46:29.614935 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.614912 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca" pod "cbace6f6-99d5-4322-a55f-ec8eba406b02" (UID: "cbace6f6-99d5-4322-a55f-ec8eba406b02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:29.616567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.616541 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca" pod "cbace6f6-99d5-4322-a55f-ec8eba406b02" (UID: "cbace6f6-99d5-4322-a55f-ec8eba406b02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:29.616707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.616690 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx" pod "cbace6f6-99d5-4322-a55f-ec8eba406b02" (UID: "cbace6f6-99d5-4322-a55f-ec8eba406b02"). InnerVolumeSpecName "kube-api-access-98nrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:29.715795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.715770 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:29.715795 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.715792 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbace6f6-99d5-4322-a55f-ec8eba406b02-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:29.715940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:29.715801 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nrx\" (UniqueName: \"kubernetes.io/projected/cbace6f6-99d5-4322-a55f-ec8eba406b02-kube-api-access-98nrx\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.114105 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:30.114061 4543 logging.go:55] [core] [Channel #32878 SubChannel #32879]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.162:50052", ServerName: "10.133.1.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.162:50052: connect: connection refused" Oct 29 22:46:30.304233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.304199 4543 generic.go:356] "Generic (PLEG): container finished" podID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerID="0cfb1ec39ade7c110fde493ca4a2c4790e10fe0308635b7d13261d0643f8565d" exitCode=0 Oct 29 22:46:30.304424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.304272 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" event={"ID":"d61ee2c6-6331-43ce-9ac3-942625eda128","Type":"ContainerDied","Data":"0cfb1ec39ade7c110fde493ca4a2c4790e10fe0308635b7d13261d0643f8565d"} Oct 29 22:46:30.306245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.306212 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" event={"ID":"cbace6f6-99d5-4322-a55f-ec8eba406b02","Type":"ContainerDied","Data":"0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462"} Oct 29 22:46:30.306365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.306255 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g" Oct 29 22:46:30.306365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.306266 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0683c8d64ebca608166d57c9dc701a3730fe2b7cb942b6127109dc52bdba0462" Oct 29 22:46:30.346319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.346271 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:46:30.423184 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.423149 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\" (UniqueName: \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\") pod \"d61ee2c6-6331-43ce-9ac3-942625eda128\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " Oct 29 22:46:30.423345 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.423188 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\") pod \"d61ee2c6-6331-43ce-9ac3-942625eda128\" (UID: \"d61ee2c6-6331-43ce-9ac3-942625eda128\") " Oct 29 22:46:30.423585 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.423564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume" pod "d61ee2c6-6331-43ce-9ac3-942625eda128" (UID: "d61ee2c6-6331-43ce-9ac3-942625eda128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:30.425526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.425502 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd" pod "d61ee2c6-6331-43ce-9ac3-942625eda128" (UID: "d61ee2c6-6331-43ce-9ac3-942625eda128"). InnerVolumeSpecName "kube-api-access-pngxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:30.524838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.524808 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngxd\" (UniqueName: \"kubernetes.io/projected/d61ee2c6-6331-43ce-9ac3-942625eda128-kube-api-access-pngxd\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.525273 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:30.524839 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61ee2c6-6331-43ce-9ac3-942625eda128-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:31.113654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.113611 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.162:50052\" within 1s: context deadline exceeded" Oct 29 22:46:31.113808 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.113698 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:46:31.270484 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.270450 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:46:31.301854 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.301827 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:46:31.314576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.314553 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" Oct 29 22:46:31.314710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.314598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn" event={"ID":"d61ee2c6-6331-43ce-9ac3-942625eda128","Type":"ContainerDied","Data":"d7672ddad5b1ef4c9b00992933559983ff310dd33f952f51fee0069c4428ce0d"} Oct 29 22:46:31.314710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.314652 4543 scope.go:122] "RemoveContainer" containerID="0cfb1ec39ade7c110fde493ca4a2c4790e10fe0308635b7d13261d0643f8565d" Oct 29 22:46:31.442781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.442711 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:46:31.451937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:31.451912 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27b6db-59d755f8db-889jn"] Oct 29 22:46:32.385550 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:32.385516 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" path="/var/lib/kubelet/pods/d61ee2c6-6331-43ce-9ac3-942625eda128/volumes" Oct 29 22:46:32.978920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:32.978878 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" containerID="cri-o://af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672" gracePeriod=25 Oct 29 22:46:33.304851 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:33.304779 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:34.028881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.028854 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:46:34.029331 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.029047 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" containerID="cri-o://acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d" gracePeriod=30 Oct 29 22:46:34.183456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183424 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt"] Oct 29 22:46:34.183954 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183941 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:34.184006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183955 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bbf528-f2f4-4ca7-8e9f-f09af5d74303" containerName="istio-proxy" Oct 29 22:46:34.184006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183969 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cbace6f6-99d5-4322-a55f-ec8eba406b02" containerName="revoke-keys" Oct 29 22:46:34.184006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183974 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbace6f6-99d5-4322-a55f-ec8eba406b02" containerName="revoke-keys" Oct 29 22:46:34.184006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.183998 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" Oct 29 22:46:34.184006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.184004 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" Oct 29 22:46:34.184197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.184111 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbace6f6-99d5-4322-a55f-ec8eba406b02" containerName="revoke-keys" Oct 29 22:46:34.184197 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.184123 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61ee2c6-6331-43ce-9ac3-942625eda128" containerName="payload-processing" Oct 29 22:46:34.188840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.188817 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.203526 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.203352 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4"] Oct 29 22:46:34.207738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207711 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:46:34.207841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207749 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:46:34.207841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207770 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:46:34.207841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207780 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:46:34.207841 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207812 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt"] Oct 29 22:46:34.208088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.207895 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.228551 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.228526 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4"] Oct 29 22:46:34.372628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372559 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.372628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372615 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.372816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372721 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2z89\" (UniqueName: \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.372816 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372758 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4989d\" (UniqueName: \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.372891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372839 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.372891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.372877 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.483595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.483554 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.483911 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.483890 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2z89\" (UniqueName: \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.484088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.484053 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4989d\" (UniqueName: \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.484248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.484232 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.484430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.484411 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.484636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.484618 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.486191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.486169 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.491050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.490988 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.507725 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.507697 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.519441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.516695 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.524613 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.524585 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4989d\" (UniqueName: \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\") pod \"maas-api-revoke-sub-853c13b1-kmgx4\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.525776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.525750 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2z89\" (UniqueName: \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\") pod \"maas-api-revoke-sub-0adbf973-nkpzt\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.527535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.527513 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:34.803537 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.803505 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:34.841644 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.841617 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4"] Oct 29 22:46:34.842580 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:34.842556 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a71bc1_6db0_46d2_bf70_82e33ba3cead.slice/crio-158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201 WatchSource:0}: Error finding container 158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201: Status 404 returned error can't find the container with id 158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201 Oct 29 22:46:34.963026 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.962992 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq"] Oct 29 22:46:34.968017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:34.967991 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c3d9a-bf2zq"] Oct 29 22:46:35.120715 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:35.120692 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt"] Oct 29 22:46:35.122246 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:35.122221 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e5d7dc_f015_427f_9472_247ac056297b.slice/crio-865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba WatchSource:0}: Error finding container 865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba: Status 404 returned error can't find the container with id 865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba Oct 29 22:46:35.343560 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:35.343527 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" event={"ID":"a8a71bc1-6db0-46d2-bf70-82e33ba3cead","Type":"ContainerStarted","Data":"c3cfc09072c2a40db27fcccaf085eb8d9a320c05fcc3ca26f9d09fe612ad0614"} Oct 29 22:46:35.343681 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:35.343583 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" event={"ID":"a8a71bc1-6db0-46d2-bf70-82e33ba3cead","Type":"ContainerStarted","Data":"158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201"} Oct 29 22:46:35.344926 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:35.344896 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" event={"ID":"19e5d7dc-f015-427f-9472-247ac056297b","Type":"ContainerStarted","Data":"865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba"} Oct 29 22:46:35.400175 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:35.400103 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" podStartSLOduration=1.400070153 podStartE2EDuration="1.400070153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:35.3981833 +0000 UTC m=+3857.140113976" watchObservedRunningTime="2026-10-29 22:46:35.400070153 +0000 UTC m=+3857.142000826" Oct 29 22:46:36.302897 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.302859 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" containerID="cri-o://07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b" gracePeriod=25 Oct 29 22:46:36.315755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.315719 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" containerID="cri-o://0535c052e3c02735eff13830054e5f5c68ab1bebf98b0bd795911aba5e67322c" gracePeriod=25 Oct 29 22:46:36.353483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.353455 4543 generic.go:356] "Generic (PLEG): container finished" podID="19e5d7dc-f015-427f-9472-247ac056297b" containerID="80599d2b67575922bc8e0d2f0d01f1e86a7879506c933f199c6976e2aa627090" exitCode=0 Oct 29 22:46:36.353598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.353493 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" event={"ID":"19e5d7dc-f015-427f-9472-247ac056297b","Type":"ContainerDied","Data":"80599d2b67575922bc8e0d2f0d01f1e86a7879506c933f199c6976e2aa627090"} Oct 29 22:46:36.355343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.355320 4543 generic.go:356] "Generic (PLEG): container finished" podID="a8a71bc1-6db0-46d2-bf70-82e33ba3cead" containerID="c3cfc09072c2a40db27fcccaf085eb8d9a320c05fcc3ca26f9d09fe612ad0614" exitCode=0 Oct 29 22:46:36.355456 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.355362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" event={"ID":"a8a71bc1-6db0-46d2-bf70-82e33ba3cead","Type":"ContainerDied","Data":"c3cfc09072c2a40db27fcccaf085eb8d9a320c05fcc3ca26f9d09fe612ad0614"} Oct 29 22:46:36.385367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.385344 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f58773-a35e-4d40-9316-fcbc178ed884" path="/var/lib/kubelet/pods/22f58773-a35e-4d40-9316-fcbc178ed884/volumes" Oct 29 22:46:36.577983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.577902 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:36.577983 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:36.577929 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:37.587859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.587837 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:37.590011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.589992 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:37.629603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629549 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\") pod \"19e5d7dc-f015-427f-9472-247ac056297b\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " Oct 29 22:46:37.629740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629618 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\" (UniqueName: \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\") pod \"19e5d7dc-f015-427f-9472-247ac056297b\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " Oct 29 22:46:37.629740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629672 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\") pod \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " Oct 29 22:46:37.629740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\" (UniqueName: \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\") pod \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " Oct 29 22:46:37.630210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\") pod \"19e5d7dc-f015-427f-9472-247ac056297b\" (UID: \"19e5d7dc-f015-427f-9472-247ac056297b\") " Oct 29 22:46:37.630210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\") pod \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\" (UID: \"a8a71bc1-6db0-46d2-bf70-82e33ba3cead\") " Oct 29 22:46:37.630210 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.629935 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca" pod "19e5d7dc-f015-427f-9472-247ac056297b" (UID: "19e5d7dc-f015-427f-9472-247ac056297b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:37.630448 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.630428 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.630527 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.630499 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca" pod "a8a71bc1-6db0-46d2-bf70-82e33ba3cead" (UID: "a8a71bc1-6db0-46d2-bf70-82e33ba3cead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:37.633127 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.633101 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d" pod "a8a71bc1-6db0-46d2-bf70-82e33ba3cead" (UID: "a8a71bc1-6db0-46d2-bf70-82e33ba3cead"). InnerVolumeSpecName "kube-api-access-4989d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.633338 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.633311 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca" pod "a8a71bc1-6db0-46d2-bf70-82e33ba3cead" (UID: "a8a71bc1-6db0-46d2-bf70-82e33ba3cead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.633718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.633691 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89" pod "19e5d7dc-f015-427f-9472-247ac056297b" (UID: "19e5d7dc-f015-427f-9472-247ac056297b"). InnerVolumeSpecName "kube-api-access-r2z89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.633976 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.633960 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca" pod "19e5d7dc-f015-427f-9472-247ac056297b" (UID: "19e5d7dc-f015-427f-9472-247ac056297b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.731862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.731824 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2z89\" (UniqueName: \"kubernetes.io/projected/19e5d7dc-f015-427f-9472-247ac056297b-kube-api-access-r2z89\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.731862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.731855 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.731862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.731865 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4989d\" (UniqueName: \"kubernetes.io/projected/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-kube-api-access-4989d\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.732103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.731876 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5d7dc-f015-427f-9472-247ac056297b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.732103 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.731886 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71bc1-6db0-46d2-bf70-82e33ba3cead-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.958939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.958861 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:37.958939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:37.958921 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:38.175575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.175542 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g"] Oct 29 22:46:38.178665 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.178636 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-580af2-7bc7g"] Oct 29 22:46:38.372801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.372774 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" Oct 29 22:46:38.372991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.372804 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853c13b1-kmgx4" event={"ID":"a8a71bc1-6db0-46d2-bf70-82e33ba3cead","Type":"ContainerDied","Data":"158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201"} Oct 29 22:46:38.372991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.372855 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158b1d2593802bd7debdce72d9a1efd3fde0fb74b0ad4cee6ee08ccde9341201" Oct 29 22:46:38.374351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.374330 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" Oct 29 22:46:38.374516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.374361 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adbf973-nkpzt" event={"ID":"19e5d7dc-f015-427f-9472-247ac056297b","Type":"ContainerDied","Data":"865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba"} Oct 29 22:46:38.374516 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.374399 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="865009a72402b543866e888b21d50b8a58419f9feb26e0203c4e79924d9248ba" Oct 29 22:46:38.386772 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:38.386748 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbace6f6-99d5-4322-a55f-ec8eba406b02" path="/var/lib/kubelet/pods/cbace6f6-99d5-4322-a55f-ec8eba406b02/volumes" Oct 29 22:46:39.299940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.299883 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:46:39.349092 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349049 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349138 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349166 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349233 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349260 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349292 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349314 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349405 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349472 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349435 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:39.349630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349472 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\") pod \"11c04983-e642-4e28-b94c-20474c7ae6bf\" (UID: \"11c04983-e642-4e28-b94c-20474c7ae6bf\") " Oct 29 22:46:39.349630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349600 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:39.349738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349661 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:39.349893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349868 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:39.349893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.349884 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:39.350340 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.350318 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c04983-e642-4e28-b94c-20474c7ae6bf-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.350455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.350347 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.350455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.350362 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.350455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.350376 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.350455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.350389 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.351575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.351553 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:39.351575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.351569 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:39.351886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.351642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "kube-api-access-5hxsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:39.351886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.351743 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo" pod "11c04983-e642-4e28-b94c-20474c7ae6bf" (UID: "11c04983-e642-4e28-b94c-20474c7ae6bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:39.384215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.384182 4543 generic.go:356] "Generic (PLEG): container finished" podID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerID="acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d" exitCode=0 Oct 29 22:46:39.384320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.384225 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" Oct 29 22:46:39.384320 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.384268 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" event={"ID":"11c04983-e642-4e28-b94c-20474c7ae6bf","Type":"ContainerDied","Data":"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d"} Oct 29 22:46:39.384415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.384331 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp" event={"ID":"11c04983-e642-4e28-b94c-20474c7ae6bf","Type":"ContainerDied","Data":"90fcdfba149bb0bdbc1266cee0aed34ce00026d1e20777a04b87fa9aac8a4ad6"} Oct 29 22:46:39.384415 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.384361 4543 scope.go:122] "RemoveContainer" containerID="acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d" Oct 29 22:46:39.396191 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.396175 4543 scope.go:122] "RemoveContainer" containerID="acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d" Oct 29 22:46:39.396457 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:39.396439 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d\": container with ID starting with acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d not found: ID does not exist" containerID="acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d" Oct 29 22:46:39.396535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.396468 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d"} err="failed to get container status \"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d\": rpc error: code = NotFound desc = could not find container \"acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d\": container with ID starting with acb2971c874cb70d0d9becb59f729fbb90ab091c772dc87822481d81a007514d not found: ID does not exist" Oct 29 22:46:39.444180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.444158 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:46:39.451468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.451446 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.451599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.451477 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.451599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.451491 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxsr\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-kube-api-access-5hxsr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.451599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.451506 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c04983-e642-4e28-b94c-20474c7ae6bf-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:39.454561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:39.454541 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-1e74c3-openshift-default-cb8557f-7fwgp"] Oct 29 22:46:40.385194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:40.385160 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" path="/var/lib/kubelet/pods/11c04983-e642-4e28-b94c-20474c7ae6bf/volumes" Oct 29 22:46:43.304956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.304901 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:43.337629 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.337598 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:46:43.337819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.337799 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" containerID="cri-o://3ead69d8712c586286ed67f61555d4da9f984abb9d4c1462da26878622149e4b" gracePeriod=30 Oct 29 22:46:43.929737 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.929705 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh"] Oct 29 22:46:43.930250 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930236 4543 cpu_manager.go:414] "containerMap: removing container" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" Oct 29 22:46:43.930304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930251 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" Oct 29 22:46:43.930304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930273 4543 cpu_manager.go:414] "containerMap: removing container" podUID="19e5d7dc-f015-427f-9472-247ac056297b" containerName="revoke-keys" Oct 29 22:46:43.930304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930279 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e5d7dc-f015-427f-9472-247ac056297b" containerName="revoke-keys" Oct 29 22:46:43.930304 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930301 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a8a71bc1-6db0-46d2-bf70-82e33ba3cead" containerName="revoke-keys" Oct 29 22:46:43.930433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930307 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a71bc1-6db0-46d2-bf70-82e33ba3cead" containerName="revoke-keys" Oct 29 22:46:43.930433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930402 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c04983-e642-4e28-b94c-20474c7ae6bf" containerName="istio-proxy" Oct 29 22:46:43.930433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930414 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e5d7dc-f015-427f-9472-247ac056297b" containerName="revoke-keys" Oct 29 22:46:43.930433 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.930424 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a71bc1-6db0-46d2-bf70-82e33ba3cead" containerName="revoke-keys" Oct 29 22:46:43.934052 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.934035 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:43.953986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.953963 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh"] Oct 29 22:46:43.963439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.963419 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:46:43.963439 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.963419 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:46:43.963577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.963450 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:46:43.964063 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.964047 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:46:43.995391 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.995366 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:43.995525 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.995416 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ffl\" (UniqueName: \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:43.995573 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:43.995518 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.096289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.096270 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.096383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.096325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ffl\" (UniqueName: \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.096383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.096369 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.097055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.096989 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.098811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.098784 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.132593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.132556 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ffl\" (UniqueName: \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\") pod \"maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.248336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.248308 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:44.582695 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:44.582666 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh"] Oct 29 22:46:44.584449 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:44.584421 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d6accf_d895_4d84_8e47_c07bc7f3cfab.slice/crio-5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df WatchSource:0}: Error finding container 5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df: Status 404 returned error can't find the container with id 5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df Oct 29 22:46:45.488149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:45.488119 4543 generic.go:356] "Generic (PLEG): container finished" podID="f6d6accf-d895-4d84-8e47-c07bc7f3cfab" containerID="3795da495c23de3e21ae45a41605eebcf237934b2556226af2e8938614ec27f8" exitCode=0 Oct 29 22:46:45.488277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:45.488201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" event={"ID":"f6d6accf-d895-4d84-8e47-c07bc7f3cfab","Type":"ContainerDied","Data":"3795da495c23de3e21ae45a41605eebcf237934b2556226af2e8938614ec27f8"} Oct 29 22:46:45.488277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:45.488273 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" event={"ID":"f6d6accf-d895-4d84-8e47-c07bc7f3cfab","Type":"ContainerStarted","Data":"5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df"} Oct 29 22:46:46.577467 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.577434 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:46.578123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.577504 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:46.645424 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.645402 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:46.727670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.727641 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\" (UniqueName: \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\") pod \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " Oct 29 22:46:46.727801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.727720 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\") pod \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " Oct 29 22:46:46.727801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.727784 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\") pod \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\" (UID: \"f6d6accf-d895-4d84-8e47-c07bc7f3cfab\") " Oct 29 22:46:46.728161 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.728132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca" pod "f6d6accf-d895-4d84-8e47-c07bc7f3cfab" (UID: "f6d6accf-d895-4d84-8e47-c07bc7f3cfab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:46.728312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.728290 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:46.729951 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.729927 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca" pod "f6d6accf-d895-4d84-8e47-c07bc7f3cfab" (UID: "f6d6accf-d895-4d84-8e47-c07bc7f3cfab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:46.730053 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.730008 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl" pod "f6d6accf-d895-4d84-8e47-c07bc7f3cfab" (UID: "f6d6accf-d895-4d84-8e47-c07bc7f3cfab"). InnerVolumeSpecName "kube-api-access-q4ffl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:46.829253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.829201 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4ffl\" (UniqueName: \"kubernetes.io/projected/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-kube-api-access-q4ffl\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:46.829253 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:46.829225 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d6accf-d895-4d84-8e47-c07bc7f3cfab-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:47.507317 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:47.507290 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" Oct 29 22:46:47.507496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:47.507290 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh" event={"ID":"f6d6accf-d895-4d84-8e47-c07bc7f3cfab","Type":"ContainerDied","Data":"5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df"} Oct 29 22:46:47.507496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:47.507418 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5080f2c49c8348856eb823eb33f21f7e8568217eaf7d17b08b975ec6c364d7df" Oct 29 22:46:47.951188 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:47.951106 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:46:47.951610 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:47.951413 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" containerID="cri-o://63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff" gracePeriod=30 Oct 29 22:46:48.515587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.515556 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp"] Oct 29 22:46:48.516475 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.516452 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f6d6accf-d895-4d84-8e47-c07bc7f3cfab" containerName="revoke-keys" Oct 29 22:46:48.516618 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.516605 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d6accf-d895-4d84-8e47-c07bc7f3cfab" containerName="revoke-keys" Oct 29 22:46:48.516892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.516880 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d6accf-d895-4d84-8e47-c07bc7f3cfab" containerName="revoke-keys" Oct 29 22:46:48.521393 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.521365 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.521743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.521719 4543 generic.go:356] "Generic (PLEG): container finished" podID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerID="3ead69d8712c586286ed67f61555d4da9f984abb9d4c1462da26878622149e4b" exitCode=0 Oct 29 22:46:48.522011 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.521986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerDied","Data":"3ead69d8712c586286ed67f61555d4da9f984abb9d4c1462da26878622149e4b"} Oct 29 22:46:48.522123 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.522049 4543 scope.go:122] "RemoveContainer" containerID="7f3ae20bebfb73974c30d35f178ef3232709feddae783fd00a67f4f1225a38c2" Oct 29 22:46:48.539170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.539128 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp"] Oct 29 22:46:48.547166 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.547137 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:46:48.547288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.547179 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:46:48.602167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.602145 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:46:48.633377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.633323 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:46:48.633377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.633338 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:46:48.657609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657589 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657709 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657674 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657705 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657773 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657725 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657801 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.657891 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657855 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.658051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657893 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.658051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658000 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\") pod \"877914d6-a5e3-4201-861f-52de61c9d9ff\" (UID: \"877914d6-a5e3-4201-861f-52de61c9d9ff\") " Oct 29 22:46:48.658208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.657891 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:48.658208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658127 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:48.658208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658061 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:48.658208 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658153 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:48.658377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658318 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.658377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658335 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:48.658532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658507 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.658678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.658655 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlgm\" (UniqueName: \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.659038 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.659019 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.659126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.659045 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877914d6-a5e3-4201-861f-52de61c9d9ff-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.659126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.659060 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.659126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.659097 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.659126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.659111 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.660884 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.660824 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:48.661458 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.661435 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "kube-api-access-gx5xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:48.661748 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.661701 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:48.663389 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.663363 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token" pod "877914d6-a5e3-4201-861f-52de61c9d9ff" (UID: "877914d6-a5e3-4201-861f-52de61c9d9ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:48.759947 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.759925 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.760112 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760037 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.760186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760157 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlgm\" (UniqueName: \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.760261 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760241 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5xr\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-kube-api-access-gx5xr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.760325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760263 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.760325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760277 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.760325 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760291 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877914d6-a5e3-4201-861f-52de61c9d9ff-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:48.760700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.760681 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.762374 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.762356 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.788918 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.788891 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlgm\" (UniqueName: \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\") pod \"maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:48.840643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:48.840623 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:49.149676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.149651 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp"] Oct 29 22:46:49.150760 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:49.150734 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f9ec7f_0251_4ee6_a282_6c928057e28b.slice/crio-290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7 WatchSource:0}: Error finding container 290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7: Status 404 returned error can't find the container with id 290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7 Oct 29 22:46:49.530145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.530119 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" Oct 29 22:46:49.530145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.530131 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng" event={"ID":"877914d6-a5e3-4201-861f-52de61c9d9ff","Type":"ContainerDied","Data":"009253fabb2bb2db7cabf3d20fb404cca99a48053625cb7fcada2ff7fe9893f8"} Oct 29 22:46:49.530301 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.530184 4543 scope.go:122] "RemoveContainer" containerID="3ead69d8712c586286ed67f61555d4da9f984abb9d4c1462da26878622149e4b" Oct 29 22:46:49.553886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.539018 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" event={"ID":"53f9ec7f-0251-4ee6-a282-6c928057e28b","Type":"ContainerStarted","Data":"b8e0bcbdd74d5fb064575bfbabeb390ddc0c8f628c727b2b06b0379c576d5b85"} Oct 29 22:46:49.553886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.540221 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" event={"ID":"53f9ec7f-0251-4ee6-a282-6c928057e28b","Type":"ContainerStarted","Data":"290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7"} Oct 29 22:46:49.603740 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.603700 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:46:49.617056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.617026 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c3d9a-openshift-default-6c799db9c7-r8kng"] Oct 29 22:46:49.650486 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.650443 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" podStartSLOduration=1.650430099 podStartE2EDuration="1.650430099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:49.649644517 +0000 UTC m=+3871.391575189" watchObservedRunningTime="2026-10-29 22:46:49.650430099 +0000 UTC m=+3871.392360773" Oct 29 22:46:49.865940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.865905 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd"] Oct 29 22:46:49.869905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:49.869879 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3b067-2mzqd"] Oct 29 22:46:50.025378 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:50.025350 4543 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f9ec7f_0251_4ee6_a282_6c928057e28b.slice/crio-conmon-b8e0bcbdd74d5fb064575bfbabeb390ddc0c8f628c727b2b06b0379c576d5b85.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:46:50.385781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:50.385755 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" path="/var/lib/kubelet/pods/877914d6-a5e3-4201-861f-52de61c9d9ff/volumes" Oct 29 22:46:50.386214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:50.386198 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeff3914-9805-4645-831c-a7fd796a4787" path="/var/lib/kubelet/pods/aeff3914-9805-4645-831c-a7fd796a4787/volumes" Oct 29 22:46:50.548245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:50.548221 4543 generic.go:356] "Generic (PLEG): container finished" podID="53f9ec7f-0251-4ee6-a282-6c928057e28b" containerID="b8e0bcbdd74d5fb064575bfbabeb390ddc0c8f628c727b2b06b0379c576d5b85" exitCode=0 Oct 29 22:46:50.548379 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:50.548300 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" event={"ID":"53f9ec7f-0251-4ee6-a282-6c928057e28b","Type":"ContainerDied","Data":"b8e0bcbdd74d5fb064575bfbabeb390ddc0c8f628c727b2b06b0379c576d5b85"} Oct 29 22:46:51.744303 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.744280 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:51.903023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.902945 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\") pod \"53f9ec7f-0251-4ee6-a282-6c928057e28b\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " Oct 29 22:46:51.903023 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.903002 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\") pod \"53f9ec7f-0251-4ee6-a282-6c928057e28b\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " Oct 29 22:46:51.903300 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.903057 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\" (UniqueName: \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\") pod \"53f9ec7f-0251-4ee6-a282-6c928057e28b\" (UID: \"53f9ec7f-0251-4ee6-a282-6c928057e28b\") " Oct 29 22:46:51.903434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.903409 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca" pod "53f9ec7f-0251-4ee6-a282-6c928057e28b" (UID: "53f9ec7f-0251-4ee6-a282-6c928057e28b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:51.905335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.905311 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm" pod "53f9ec7f-0251-4ee6-a282-6c928057e28b" (UID: "53f9ec7f-0251-4ee6-a282-6c928057e28b"). InnerVolumeSpecName "kube-api-access-tvlgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:51.905414 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:51.905334 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca" pod "53f9ec7f-0251-4ee6-a282-6c928057e28b" (UID: "53f9ec7f-0251-4ee6-a282-6c928057e28b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:52.004565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.004542 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:52.004565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.004565 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f9ec7f-0251-4ee6-a282-6c928057e28b-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:52.004706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.004575 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlgm\" (UniqueName: \"kubernetes.io/projected/53f9ec7f-0251-4ee6-a282-6c928057e28b-kube-api-access-tvlgm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:52.566436 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.566403 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" event={"ID":"53f9ec7f-0251-4ee6-a282-6c928057e28b","Type":"ContainerDied","Data":"290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7"} Oct 29 22:46:52.566587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.566453 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp" Oct 29 22:46:52.566587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:52.566459 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290417cc541609e6334ea7385c1bde262987cc973cea7ab31586d684d96ff2b7" Oct 29 22:46:53.244962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.244940 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:46:53.272558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.272532 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 22:46:53.420443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420391 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\") pod \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " Oct 29 22:46:53.420443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420441 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420526 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420569 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\" (UniqueName: \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\") pod \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\" (UID: \"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420618 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420674 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.420756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420716 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.421160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420825 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.421160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.421160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.420899 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\") pod \"f253a666-b1cf-48a6-ad54-41276739b5b8\" (UID: \"f253a666-b1cf-48a6-ad54-41276739b5b8\") " Oct 29 22:46:53.421643 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.421614 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.422307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.422245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.422457 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.422430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.423849 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.423806 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.424106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.424069 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume" pod "5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" (UID: "5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.424327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.424305 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:53.424431 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.424357 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.424874 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.424852 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "kube-api-access-fcn7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.425777 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.425753 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.425882 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.425858 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy" pod "f253a666-b1cf-48a6-ad54-41276739b5b8" (UID: "f253a666-b1cf-48a6-ad54-41276739b5b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.426158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.426138 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6" pod "5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" (UID: "5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7"). InnerVolumeSpecName "kube-api-access-dnjm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.521941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521888 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.521941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521914 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnjm6\" (UniqueName: \"kubernetes.io/projected/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-kube-api-access-dnjm6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.521941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521924 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcn7w\" (UniqueName: \"kubernetes.io/projected/f253a666-b1cf-48a6-ad54-41276739b5b8-kube-api-access-fcn7w\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.521941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521933 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.521941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521942 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521951 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f253a666-b1cf-48a6-ad54-41276739b5b8-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521959 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521969 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f253a666-b1cf-48a6-ad54-41276739b5b8-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521982 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.521992 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.522272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.522001 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f253a666-b1cf-48a6-ad54-41276739b5b8-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.574732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.574709 4543 generic.go:356] "Generic (PLEG): container finished" podID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerID="63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff" exitCode=0 Oct 29 22:46:53.574847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.574765 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" Oct 29 22:46:53.574847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.574800 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" event={"ID":"f253a666-b1cf-48a6-ad54-41276739b5b8","Type":"ContainerDied","Data":"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff"} Oct 29 22:46:53.574847 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.574847 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw" event={"ID":"f253a666-b1cf-48a6-ad54-41276739b5b8","Type":"ContainerDied","Data":"fec8176eb601891fedfdbe76329f491f302550937bef1b6f54c848ddce9097e0"} Oct 29 22:46:53.575025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.574868 4543 scope.go:122] "RemoveContainer" containerID="63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff" Oct 29 22:46:53.576438 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.576411 4543 generic.go:356] "Generic (PLEG): container finished" podID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerID="af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672" exitCode=0 Oct 29 22:46:53.576555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.576456 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" Oct 29 22:46:53.576555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.576492 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" event={"ID":"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7","Type":"ContainerDied","Data":"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672"} Oct 29 22:46:53.576555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.576543 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg" event={"ID":"5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7","Type":"ContainerDied","Data":"7fc530f547159d238724fd409e9c0b5a1053dee81921b809800793487d26244c"} Oct 29 22:46:53.587157 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.587119 4543 scope.go:122] "RemoveContainer" containerID="63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff" Oct 29 22:46:53.587482 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:53.587453 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff\": container with ID starting with 63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff not found: ID does not exist" containerID="63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff" Oct 29 22:46:53.587577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.587488 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff"} err="failed to get container status \"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff\": rpc error: code = NotFound desc = could not find container \"63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff\": container with ID starting with 63c304059dba71af44d074533181a89bd2158186a4f538fabf7a02159e37e0ff not found: ID does not exist" Oct 29 22:46:53.587577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.587504 4543 scope.go:122] "RemoveContainer" containerID="af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672" Oct 29 22:46:53.607496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.607465 4543 scope.go:122] "RemoveContainer" containerID="af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672" Oct 29 22:46:53.607784 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:53.607765 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672\": container with ID starting with af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672 not found: ID does not exist" containerID="af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672" Oct 29 22:46:53.607843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.607790 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672"} err="failed to get container status \"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672\": rpc error: code = NotFound desc = could not find container \"af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672\": container with ID starting with af069142bd1b1370fcad7096cb003e577c7d5b74bf3344c74910e1880cd10672 not found: ID does not exist" Oct 29 22:46:53.635403 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.635381 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:46:53.648385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.648363 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-r6hzw"] Oct 29 22:46:53.680444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.680422 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 22:46:53.688029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:53.688009 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3b067-65df4bdfb6-lpnhg"] Oct 29 22:46:54.384599 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:54.384529 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" path="/var/lib/kubelet/pods/5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7/volumes" Oct 29 22:46:54.385014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:54.384988 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" path="/var/lib/kubelet/pods/f253a666-b1cf-48a6-ad54-41276739b5b8/volumes" Oct 29 22:46:54.428829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:54.428801 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:46:54.464126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:54.464105 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:46:56.567005 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.566936 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:46:56.607802 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.607775 4543 generic.go:356] "Generic (PLEG): container finished" podID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerID="0535c052e3c02735eff13830054e5f5c68ab1bebf98b0bd795911aba5e67322c" exitCode=0 Oct 29 22:46:56.607925 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.607851 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" event={"ID":"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3","Type":"ContainerDied","Data":"0535c052e3c02735eff13830054e5f5c68ab1bebf98b0bd795911aba5e67322c"} Oct 29 22:46:56.609520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.609474 4543 generic.go:356] "Generic (PLEG): container finished" podID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerID="07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b" exitCode=0 Oct 29 22:46:56.609693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.609524 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" event={"ID":"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f","Type":"ContainerDied","Data":"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b"} Oct 29 22:46:56.609693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.609539 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" Oct 29 22:46:56.609693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.609564 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8" event={"ID":"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f","Type":"ContainerDied","Data":"28f56f6ea322094b01fd6a842df334e6c66bf75b6d92772cd639c2800b2177dc"} Oct 29 22:46:56.609693 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.609592 4543 scope.go:122] "RemoveContainer" containerID="07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b" Oct 29 22:46:56.640444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.640412 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:46:56.646804 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.646784 4543 scope.go:122] "RemoveContainer" containerID="07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b" Oct 29 22:46:56.647096 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:56.647061 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b\": container with ID starting with 07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b not found: ID does not exist" containerID="07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b" Oct 29 22:46:56.647202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.647108 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b"} err="failed to get container status \"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b\": rpc error: code = NotFound desc = could not find container \"07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b\": container with ID starting with 07ed4b54c12b7a1835698d47c8ff40352cec867dc272bb1525046e73673ff24b not found: ID does not exist" Oct 29 22:46:56.753690 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.753660 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\") pod \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " Oct 29 22:46:56.753845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.753782 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\" (UniqueName: \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\") pod \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\" (UID: \"10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f\") " Oct 29 22:46:56.753845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.753806 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\" (UniqueName: \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\") pod \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " Oct 29 22:46:56.753845 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.753832 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\") pod \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\" (UID: \"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3\") " Oct 29 22:46:56.754131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.754106 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume" pod "10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" (UID: "10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:56.754318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.754299 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume" pod "2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" (UID: "2cb15e08-2fff-48d7-87f0-1b0f6de25ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:56.756032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.756014 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7" pod "10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" (UID: "10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f"). InnerVolumeSpecName "kube-api-access-wq8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.756126 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.756030 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh" pod "2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" (UID: "2cb15e08-2fff-48d7-87f0-1b0f6de25ba3"). InnerVolumeSpecName "kube-api-access-gx5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.854778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.854720 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.854778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.854744 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.854778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.854757 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq8j7\" (UniqueName: \"kubernetes.io/projected/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f-kube-api-access-wq8j7\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.854778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.854766 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5vh\" (UniqueName: \"kubernetes.io/projected/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3-kube-api-access-gx5vh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.972913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.972877 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:46:56.985776 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:56.985739 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c3d9a-6df69d5f9-nxtr8"] Oct 29 22:46:57.551824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.551760 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 22:46:57.552018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.551950 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" containerID="cri-o://b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10" gracePeriod=30 Oct 29 22:46:57.577453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.577384 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.58:50052\" within 1s: context deadline exceeded" Oct 29 22:46:57.577453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.577455 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:46:57.577453 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:46:57.577456 4543 logging.go:55] [core] [Channel #32991 SubChannel #32992]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.58:50052", ServerName: "10.133.1.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.58:50052: operation was canceled" Oct 29 22:46:57.617859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.617833 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" Oct 29 22:46:57.617987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.617877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k" event={"ID":"2cb15e08-2fff-48d7-87f0-1b0f6de25ba3","Type":"ContainerDied","Data":"af7403afde46442df15cdcd28f3fd35a759b9ce973a9ece2899ab9768996c4d2"} Oct 29 22:46:57.617987 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.617929 4543 scope.go:122] "RemoveContainer" containerID="0535c052e3c02735eff13830054e5f5c68ab1bebf98b0bd795911aba5e67322c" Oct 29 22:46:57.685704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.685663 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:46:57.702732 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:57.702700 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c3d9a-d8ff9c596-8h68k"] Oct 29 22:46:57.959884 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:46:57.959801 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:46:58.390915 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:58.390873 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" path="/var/lib/kubelet/pods/10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f/volumes" Oct 29 22:46:58.391536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:58.391506 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" path="/var/lib/kubelet/pods/2cb15e08-2fff-48d7-87f0-1b0f6de25ba3/volumes" Oct 29 22:46:59.053471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:59.053433 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:46:59.092917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:59.092881 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:46:59.429945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:59.429842 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" containerID="cri-o://e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965" gracePeriod=25 Oct 29 22:46:59.464365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:46:59.464326 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" containerID="cri-o://1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0" gracePeriod=25 Oct 29 22:47:00.011243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:00.011198 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:00.011466 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:00.011214 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:01.405370 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:01.405335 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:47:01.405749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:01.405404 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:47:02.815812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.815775 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 22:47:02.921248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921180 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921270 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921340 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921378 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921439 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921474 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921541 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921615 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921655 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\") pod \"1c07798b-4181-4248-8f40-78a86dab08d4\" (UID: \"1c07798b-4181-4248-8f40-78a86dab08d4\") " Oct 29 22:47:02.921785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921657 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.921950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921805 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.921950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.921895 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.922091 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922021 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.922235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922212 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.922307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922289 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.922372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922307 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.922372 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922322 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.922632 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.922388 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:02.923662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.923640 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:02.924145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.924117 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.924408 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.924384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "kube-api-access-tbp2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:02.924590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:02.924570 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo" pod "1c07798b-4181-4248-8f40-78a86dab08d4" (UID: "1c07798b-4181-4248-8f40-78a86dab08d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:03.023558 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.023538 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c07798b-4181-4248-8f40-78a86dab08d4-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.023676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.023560 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbp2r\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-kube-api-access-tbp2r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.023676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.023571 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c07798b-4181-4248-8f40-78a86dab08d4-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.023676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.023579 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c07798b-4181-4248-8f40-78a86dab08d4-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.023676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.023588 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c07798b-4181-4248-8f40-78a86dab08d4-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.669754 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.669716 4543 generic.go:356] "Generic (PLEG): container finished" podID="1c07798b-4181-4248-8f40-78a86dab08d4" containerID="b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10" exitCode=0 Oct 29 22:47:03.669962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.669763 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" event={"ID":"1c07798b-4181-4248-8f40-78a86dab08d4","Type":"ContainerDied","Data":"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10"} Oct 29 22:47:03.669962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.669774 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" Oct 29 22:47:03.669962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.669843 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9" event={"ID":"1c07798b-4181-4248-8f40-78a86dab08d4","Type":"ContainerDied","Data":"c6732f57b545d0365ddc61a3c92ee6b88b2f6fd43d4b61ac9041a5bc1dbf7c77"} Oct 29 22:47:03.669962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.669873 4543 scope.go:122] "RemoveContainer" containerID="b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10" Oct 29 22:47:03.681125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.681107 4543 scope.go:122] "RemoveContainer" containerID="b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10" Oct 29 22:47:03.681406 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:03.681387 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10\": container with ID starting with b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10 not found: ID does not exist" containerID="b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10" Oct 29 22:47:03.681451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.681414 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10"} err="failed to get container status \"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10\": rpc error: code = NotFound desc = could not find container \"b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10\": container with ID starting with b0edff8e109de7a2e0041184a5d104229a237f81ce8f31698a2287a802bc4a10 not found: ID does not exist" Oct 29 22:47:03.730624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.730602 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 22:47:03.743769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:03.743744 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-2m5t9"] Oct 29 22:47:04.054658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:04.054611 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" containerID="cri-o://2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846" gracePeriod=25 Oct 29 22:47:04.093815 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:04.093756 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" containerID="cri-o://9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef" gracePeriod=25 Oct 29 22:47:04.384420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:04.384363 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" path="/var/lib/kubelet/pods/1c07798b-4181-4248-8f40-78a86dab08d4/volumes" Oct 29 22:47:04.693757 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:04.693678 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:05.256346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:05.256313 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp"] Oct 29 22:47:05.269406 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:05.269379 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-749f72-wnjqp"] Oct 29 22:47:05.792668 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:05.792623 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh"] Oct 29 22:47:05.815120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:05.812321 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-07e166-w7zjh"] Oct 29 22:47:06.386487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:06.386457 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f9ec7f-0251-4ee6-a282-6c928057e28b" path="/var/lib/kubelet/pods/53f9ec7f-0251-4ee6-a282-6c928057e28b/volumes" Oct 29 22:47:06.386877 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:06.386862 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d6accf-d895-4d84-8e47-c07bc7f3cfab" path="/var/lib/kubelet/pods/f6d6accf-d895-4d84-8e47-c07bc7f3cfab/volumes" Oct 29 22:47:10.009655 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:10.009613 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:10.010054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:10.009616 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:13.684986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:13.684948 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:14.692647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.692611 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:14.928059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928025 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx"] Oct 29 22:47:14.928571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928554 4543 cpu_manager.go:414] "containerMap: removing container" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" Oct 29 22:47:14.928571 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928571 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928589 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928595 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928611 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928616 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928629 4543 cpu_manager.go:414] "containerMap: removing container" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928634 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928640 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928644 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928653 4543 cpu_manager.go:414] "containerMap: removing container" podUID="53f9ec7f-0251-4ee6-a282-6c928057e28b" containerName="revoke-keys" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928657 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f9ec7f-0251-4ee6-a282-6c928057e28b" containerName="revoke-keys" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928666 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928670 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928678 4543 cpu_manager.go:414] "containerMap: removing container" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:14.928729 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928683 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928754 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c07798b-4181-4248-8f40-78a86dab08d4" containerName="istio-proxy" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928763 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f9ec7f-0251-4ee6-a282-6c928057e28b" containerName="revoke-keys" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928769 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f253a666-b1cf-48a6-ad54-41276739b5b8" containerName="istio-proxy" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928773 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928781 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b9d7c8-9ed4-4c74-a9c2-0bf1f638769f" containerName="payload-pre-processing" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928789 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6ea3c0-2aa9-42d2-a097-3570b9ebb0a7" containerName="payload-processing" Oct 29 22:47:14.929490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.928796 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb15e08-2fff-48d7-87f0-1b0f6de25ba3" containerName="payload-processing" Oct 29 22:47:14.933979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.933961 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:14.952649 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.952590 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx"] Oct 29 22:47:14.961419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.961393 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:47:14.961532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.961397 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:47:14.979779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.979750 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:47:14.979889 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:14.979853 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:47:15.043288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.043267 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rj87\" (UniqueName: \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.043398 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.043317 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.043462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.043443 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.144927 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.144891 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rj87\" (UniqueName: \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.145025 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.144974 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.145098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.145056 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.145712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.145689 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.147473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.147446 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.172041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.172020 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rj87\" (UniqueName: \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\") pod \"maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.224067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.224041 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc"] Oct 29 22:47:15.224605 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.224593 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="877914d6-a5e3-4201-861f-52de61c9d9ff" containerName="istio-proxy" Oct 29 22:47:15.227980 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.227966 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.248160 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.248137 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc"] Oct 29 22:47:15.264027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.264011 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:15.346272 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.346242 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.346405 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.346311 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.346453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.346433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdvs\" (UniqueName: \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.447269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.447236 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.447417 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.447326 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdvs\" (UniqueName: \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.447488 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.447456 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.448116 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.448064 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.449713 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.449693 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.473594 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.473575 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdvs\" (UniqueName: \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\") pod \"maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.541354 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.541308 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:15.571617 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.571590 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx"] Oct 29 22:47:15.573706 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:47:15.573676 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65663dd_47a0_4a7c_bf68_89672a384efa.slice/crio-e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4 WatchSource:0}: Error finding container e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4: Status 404 returned error can't find the container with id e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4 Oct 29 22:47:15.762964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.762929 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" event={"ID":"d65663dd-47a0-4a7c-bf68-89672a384efa","Type":"ContainerStarted","Data":"e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4"} Oct 29 22:47:15.852478 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:15.852451 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc"] Oct 29 22:47:15.854508 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:47:15.854485 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784c993f_9ccd_4909_b4fd_2473476582fb.slice/crio-e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684 WatchSource:0}: Error finding container e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684: Status 404 returned error can't find the container with id e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684 Oct 29 22:47:16.771445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:16.771409 4543 generic.go:356] "Generic (PLEG): container finished" podID="784c993f-9ccd-4909-b4fd-2473476582fb" containerID="9f8c637e5d3b182b2d546c432e2f24cb8943c24cb3d7d8356c551e74780c58e2" exitCode=0 Oct 29 22:47:16.771812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:16.771493 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" event={"ID":"784c993f-9ccd-4909-b4fd-2473476582fb","Type":"ContainerDied","Data":"9f8c637e5d3b182b2d546c432e2f24cb8943c24cb3d7d8356c551e74780c58e2"} Oct 29 22:47:16.771812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:16.771544 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" event={"ID":"784c993f-9ccd-4909-b4fd-2473476582fb","Type":"ContainerStarted","Data":"e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684"} Oct 29 22:47:16.773207 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:16.773187 4543 generic.go:356] "Generic (PLEG): container finished" podID="d65663dd-47a0-4a7c-bf68-89672a384efa" containerID="0a57d15a373ecf5b33f5a2520a975af267ce2fd23a10bd02aba5213d4a1aa540" exitCode=0 Oct 29 22:47:16.773322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:16.773263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" event={"ID":"d65663dd-47a0-4a7c-bf68-89672a384efa","Type":"ContainerDied","Data":"0a57d15a373ecf5b33f5a2520a975af267ce2fd23a10bd02aba5213d4a1aa540"} Oct 29 22:47:17.948248 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:17.948216 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:17.986943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:17.986919 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:18.075394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075329 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\") pod \"784c993f-9ccd-4909-b4fd-2473476582fb\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " Oct 29 22:47:18.075547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075408 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\" (UniqueName: \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\") pod \"d65663dd-47a0-4a7c-bf68-89672a384efa\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " Oct 29 22:47:18.075547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075435 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\") pod \"d65663dd-47a0-4a7c-bf68-89672a384efa\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " Oct 29 22:47:18.075678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075548 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\") pod \"d65663dd-47a0-4a7c-bf68-89672a384efa\" (UID: \"d65663dd-47a0-4a7c-bf68-89672a384efa\") " Oct 29 22:47:18.075678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075561 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca" pod "784c993f-9ccd-4909-b4fd-2473476582fb" (UID: "784c993f-9ccd-4909-b4fd-2473476582fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:18.075678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075568 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\") pod \"784c993f-9ccd-4909-b4fd-2473476582fb\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " Oct 29 22:47:18.075857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075684 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\" (UniqueName: \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\") pod \"784c993f-9ccd-4909-b4fd-2473476582fb\" (UID: \"784c993f-9ccd-4909-b4fd-2473476582fb\") " Oct 29 22:47:18.075908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.075891 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca" pod "d65663dd-47a0-4a7c-bf68-89672a384efa" (UID: "d65663dd-47a0-4a7c-bf68-89672a384efa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:18.076444 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.076416 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.076567 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.076450 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.077840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.077811 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs" pod "784c993f-9ccd-4909-b4fd-2473476582fb" (UID: "784c993f-9ccd-4909-b4fd-2473476582fb"). InnerVolumeSpecName "kube-api-access-vsdvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:18.077840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.077830 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca" pod "d65663dd-47a0-4a7c-bf68-89672a384efa" (UID: "d65663dd-47a0-4a7c-bf68-89672a384efa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:18.078185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.078167 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca" pod "784c993f-9ccd-4909-b4fd-2473476582fb" (UID: "784c993f-9ccd-4909-b4fd-2473476582fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:18.078307 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.078195 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87" pod "d65663dd-47a0-4a7c-bf68-89672a384efa" (UID: "d65663dd-47a0-4a7c-bf68-89672a384efa"). InnerVolumeSpecName "kube-api-access-7rj87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:18.177515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.177491 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdvs\" (UniqueName: \"kubernetes.io/projected/784c993f-9ccd-4909-b4fd-2473476582fb-kube-api-access-vsdvs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.177515 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.177515 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rj87\" (UniqueName: \"kubernetes.io/projected/d65663dd-47a0-4a7c-bf68-89672a384efa-kube-api-access-7rj87\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.177656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.177525 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65663dd-47a0-4a7c-bf68-89672a384efa-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.177656 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.177535 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784c993f-9ccd-4909-b4fd-2473476582fb-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:18.789982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.789950 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" Oct 29 22:47:18.789982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.789963 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc" event={"ID":"784c993f-9ccd-4909-b4fd-2473476582fb","Type":"ContainerDied","Data":"e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684"} Oct 29 22:47:18.790211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.790024 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9167dd928cdd584131a50c3e93b0d6459f27513c8c073d9091d45897509b684" Oct 29 22:47:18.791636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.791618 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" Oct 29 22:47:18.791765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.791666 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx" event={"ID":"d65663dd-47a0-4a7c-bf68-89672a384efa","Type":"ContainerDied","Data":"e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4"} Oct 29 22:47:18.791765 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:18.791710 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e902e4103a3d7691af5f1c88d2fabad1bb295a9c06f646dfcc73a7057988c9b4" Oct 29 22:47:19.726705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.726681 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:47:19.758018 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.757994 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:47:19.790647 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.790623 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\" (UniqueName: \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\") pod \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " Oct 29 22:47:19.790790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.790767 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\") pod \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\" (UID: \"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b\") " Oct 29 22:47:19.791150 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.791128 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume" pod "937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" (UID: "937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:19.793143 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.793121 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8" pod "937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" (UID: "937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b"). InnerVolumeSpecName "kube-api-access-h22p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.804277 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.804250 4543 generic.go:356] "Generic (PLEG): container finished" podID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerID="1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0" exitCode=0 Oct 29 22:47:19.804396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.804309 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" Oct 29 22:47:19.804396 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.804347 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" event={"ID":"983248db-dedd-4c0e-9750-e4fc85e79e0b","Type":"ContainerDied","Data":"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0"} Oct 29 22:47:19.804501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.804404 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq" event={"ID":"983248db-dedd-4c0e-9750-e4fc85e79e0b","Type":"ContainerDied","Data":"71a7885ee27ca4fdab8c4a6ac7d75121dbe9ba27da5cbc4e9d2ffae6af01f6d3"} Oct 29 22:47:19.804501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.804433 4543 scope.go:122] "RemoveContainer" containerID="1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0" Oct 29 22:47:19.806309 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.806286 4543 generic.go:356] "Generic (PLEG): container finished" podID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerID="e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965" exitCode=0 Oct 29 22:47:19.806411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.806341 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" event={"ID":"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b","Type":"ContainerDied","Data":"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965"} Oct 29 22:47:19.806411 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.806379 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" event={"ID":"937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b","Type":"ContainerDied","Data":"db64e36bce6396d7f5042253c997764ed460ae29057bb4015978340518250129"} Oct 29 22:47:19.806534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.806447 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs" Oct 29 22:47:19.816780 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.816765 4543 scope.go:122] "RemoveContainer" containerID="1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0" Oct 29 22:47:19.817045 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:19.817028 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0\": container with ID starting with 1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0 not found: ID does not exist" containerID="1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0" Oct 29 22:47:19.817111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.817053 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0"} err="failed to get container status \"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0\": rpc error: code = NotFound desc = could not find container \"1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0\": container with ID starting with 1b806b092e92abacf109429cd86b4aeb7a703e5ce316f391a27e628a9c4e6db0 not found: ID does not exist" Oct 29 22:47:19.817111 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.817070 4543 scope.go:122] "RemoveContainer" containerID="e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965" Oct 29 22:47:19.828064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.828042 4543 scope.go:122] "RemoveContainer" containerID="e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965" Oct 29 22:47:19.828405 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:19.828367 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965\": container with ID starting with e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965 not found: ID does not exist" containerID="e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965" Oct 29 22:47:19.828523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.828413 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965"} err="failed to get container status \"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965\": rpc error: code = NotFound desc = could not find container \"e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965\": container with ID starting with e6081a8f03df3085cb63e1f239e687c13bad841b268963b9b2051807568a5965 not found: ID does not exist" Oct 29 22:47:19.864899 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.864835 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:47:19.869453 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.869431 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-07e166-7964dfd84-njjfs"] Oct 29 22:47:19.892242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.892220 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\" (UniqueName: \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\") pod \"983248db-dedd-4c0e-9750-e4fc85e79e0b\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " Oct 29 22:47:19.892324 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.892265 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\") pod \"983248db-dedd-4c0e-9750-e4fc85e79e0b\" (UID: \"983248db-dedd-4c0e-9750-e4fc85e79e0b\") " Oct 29 22:47:19.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.892572 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.892596 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.892590 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22p8\" (UniqueName: \"kubernetes.io/projected/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b-kube-api-access-h22p8\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.892739 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.892625 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume" pod "983248db-dedd-4c0e-9750-e4fc85e79e0b" (UID: "983248db-dedd-4c0e-9750-e4fc85e79e0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:19.894328 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.894308 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g" pod "983248db-dedd-4c0e-9750-e4fc85e79e0b" (UID: "983248db-dedd-4c0e-9750-e4fc85e79e0b"). InnerVolumeSpecName "kube-api-access-mwv9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:19.968866 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.968838 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 22:47:19.993192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.993090 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwv9g\" (UniqueName: \"kubernetes.io/projected/983248db-dedd-4c0e-9750-e4fc85e79e0b-kube-api-access-mwv9g\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:19.993192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:19.993120 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/983248db-dedd-4c0e-9750-e4fc85e79e0b-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:20.036887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.036858 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg"] Oct 29 22:47:20.043214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.043190 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65421f67-6xjcg"] Oct 29 22:47:20.159214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.159191 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:47:20.167056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.167034 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-07e166-6db66d5655-jrbtq"] Oct 29 22:47:20.384936 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.384849 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcc25e2-f3ea-429c-badf-84888839f629" path="/var/lib/kubelet/pods/8bcc25e2-f3ea-429c-badf-84888839f629/volumes" Oct 29 22:47:20.385385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.385362 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" path="/var/lib/kubelet/pods/937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b/volumes" Oct 29 22:47:20.385791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:20.385772 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" path="/var/lib/kubelet/pods/983248db-dedd-4c0e-9750-e4fc85e79e0b/volumes" Oct 29 22:47:23.684801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:23.684762 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:24.358008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.357967 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:47:24.377140 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.377111 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:47:24.431785 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.431751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\") pod \"5924a540-1317-4e5a-9251-9ced453c35be\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " Oct 29 22:47:24.431995 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.431867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\" (UniqueName: \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\") pod \"5924a540-1317-4e5a-9251-9ced453c35be\" (UID: \"5924a540-1317-4e5a-9251-9ced453c35be\") " Oct 29 22:47:24.432245 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.432218 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume" pod "5924a540-1317-4e5a-9251-9ced453c35be" (UID: "5924a540-1317-4e5a-9251-9ced453c35be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:24.434575 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.434548 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb" pod "5924a540-1317-4e5a-9251-9ced453c35be" (UID: "5924a540-1317-4e5a-9251-9ced453c35be"). InnerVolumeSpecName "kube-api-access-brmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:24.533177 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.533149 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\") pod \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " Oct 29 22:47:24.533318 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.533276 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\" (UniqueName: \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\") pod \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\" (UID: \"718bbebf-8ef8-4311-a934-7b28acfc5cfd\") " Oct 29 22:47:24.533586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.533560 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume" pod "718bbebf-8ef8-4311-a934-7b28acfc5cfd" (UID: "718bbebf-8ef8-4311-a934-7b28acfc5cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:24.533674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.533626 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5924a540-1317-4e5a-9251-9ced453c35be-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:24.533674 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.533649 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmxb\" (UniqueName: \"kubernetes.io/projected/5924a540-1317-4e5a-9251-9ced453c35be-kube-api-access-brmxb\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:24.535476 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.535458 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh" pod "718bbebf-8ef8-4311-a934-7b28acfc5cfd" (UID: "718bbebf-8ef8-4311-a934-7b28acfc5cfd"). InnerVolumeSpecName "kube-api-access-fh5lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:24.634722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.634669 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718bbebf-8ef8-4311-a934-7b28acfc5cfd-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:24.634722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.634691 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5lh\" (UniqueName: \"kubernetes.io/projected/718bbebf-8ef8-4311-a934-7b28acfc5cfd-kube-api-access-fh5lh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:24.844781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.844753 4543 generic.go:356] "Generic (PLEG): container finished" podID="5924a540-1317-4e5a-9251-9ced453c35be" containerID="2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846" exitCode=0 Oct 29 22:47:24.845200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.844803 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" Oct 29 22:47:24.845200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.844836 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" event={"ID":"5924a540-1317-4e5a-9251-9ced453c35be","Type":"ContainerDied","Data":"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846"} Oct 29 22:47:24.845200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.844889 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh" event={"ID":"5924a540-1317-4e5a-9251-9ced453c35be","Type":"ContainerDied","Data":"c77a068d236adc0749f3f88320cd187f15d4522bfc9a31e74dee4d7972c69f6b"} Oct 29 22:47:24.845200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.844918 4543 scope.go:122] "RemoveContainer" containerID="2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846" Oct 29 22:47:24.846749 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.846726 4543 generic.go:356] "Generic (PLEG): container finished" podID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerID="9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef" exitCode=0 Oct 29 22:47:24.846864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.846763 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" Oct 29 22:47:24.846864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.846799 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" event={"ID":"718bbebf-8ef8-4311-a934-7b28acfc5cfd","Type":"ContainerDied","Data":"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef"} Oct 29 22:47:24.846864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.846846 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7" event={"ID":"718bbebf-8ef8-4311-a934-7b28acfc5cfd","Type":"ContainerDied","Data":"794b4b26c343c8e78db2b428fc141a8994a45715bc14fe882ab77f07648909bf"} Oct 29 22:47:24.856831 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.856816 4543 scope.go:122] "RemoveContainer" containerID="2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846" Oct 29 22:47:24.857215 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:24.857116 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846\": container with ID starting with 2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846 not found: ID does not exist" containerID="2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846" Oct 29 22:47:24.857215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.857139 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846"} err="failed to get container status \"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846\": rpc error: code = NotFound desc = could not find container \"2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846\": container with ID starting with 2005e5d15c8fd8e08234cd6f57a47b94886bef031173e83522e96f2c6b758846 not found: ID does not exist" Oct 29 22:47:24.857215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.857155 4543 scope.go:122] "RemoveContainer" containerID="9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef" Oct 29 22:47:24.866720 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.866700 4543 scope.go:122] "RemoveContainer" containerID="9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef" Oct 29 22:47:24.866971 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:24.866952 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef\": container with ID starting with 9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef not found: ID does not exist" containerID="9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef" Oct 29 22:47:24.867050 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.866984 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef"} err="failed to get container status \"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef\": rpc error: code = NotFound desc = could not find container \"9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef\": container with ID starting with 9460beda4982680277584321307db06cbde12f658f4f56457c0beee4ed092eef not found: ID does not exist" Oct 29 22:47:24.903957 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.903904 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:47:24.916961 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.916942 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-749f72-b8c4ff79-8cds7"] Oct 29 22:47:24.955109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.955085 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:47:24.961878 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:24.961857 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-749f72-7b587788c4-z8bvh"] Oct 29 22:47:25.399810 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.399779 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 22:47:25.412243 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.412217 4543 scope.go:122] "RemoveContainer" containerID="41dfda38fe2784123d66cf9806ec15b2c0b948ebe2c78bf9cd5125ecf62753b2" Oct 29 22:47:25.422704 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.422679 4543 scope.go:122] "RemoveContainer" containerID="9776cdfdd3590519583cb4effc62979f90b9f849e8680ab2e67adab1647e9dcc" Oct 29 22:47:25.431766 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.431748 4543 scope.go:122] "RemoveContainer" containerID="51e39b1adbf88978c82763ecaa3497c55e1eb4e6d9d67477c527d7e90f862dba" Oct 29 22:47:25.440843 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.440825 4543 scope.go:122] "RemoveContainer" containerID="9adac73e35b81eea34f1ed8271ae061893efffc2cc07418a7959c0baef82669c" Oct 29 22:47:25.441941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.441922 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 22:47:25.450559 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.450541 4543 scope.go:122] "RemoveContainer" containerID="a29486058eef3eb6cd8ca2c24467103fb5e80375b605aed5179b5536e543cfc2" Oct 29 22:47:25.820657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.820629 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:47:25.845945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:25.845905 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:47:26.385132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:26.385101 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5924a540-1317-4e5a-9251-9ced453c35be" path="/var/lib/kubelet/pods/5924a540-1317-4e5a-9251-9ced453c35be/volumes" Oct 29 22:47:26.385510 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:26.385496 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" path="/var/lib/kubelet/pods/718bbebf-8ef8-4311-a934-7b28acfc5cfd/volumes" Oct 29 22:47:27.136756 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:27.136704 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx"] Oct 29 22:47:27.149120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:27.149091 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-201a3d-gwpcx"] Oct 29 22:47:27.960336 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:27.960294 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:47:28.384035 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:28.384005 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65663dd-47a0-4a7c-bf68-89672a384efa" path="/var/lib/kubelet/pods/d65663dd-47a0-4a7c-bf68-89672a384efa/volumes" Oct 29 22:47:30.400733 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.400631 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" podUID="343d315a-d25c-4d90-948b-fbc01f39558f" containerName="payload-processing" containerID="cri-o://71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2" gracePeriod=25 Oct 29 22:47:30.442238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.442193 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" podUID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" containerName="payload-pre-processing" containerID="cri-o://497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2" gracePeriod=25 Oct 29 22:47:30.692148 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.692119 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 22:47:30.730495 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.730471 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 22:47:30.799145 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.799115 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\" (UniqueName: \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\") pod \"343d315a-d25c-4d90-948b-fbc01f39558f\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " Oct 29 22:47:30.799311 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.799259 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\") pod \"343d315a-d25c-4d90-948b-fbc01f39558f\" (UID: \"343d315a-d25c-4d90-948b-fbc01f39558f\") " Oct 29 22:47:30.799697 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.799662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume" pod "343d315a-d25c-4d90-948b-fbc01f39558f" (UID: "343d315a-d25c-4d90-948b-fbc01f39558f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:30.801462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.801437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz" pod "343d315a-d25c-4d90-948b-fbc01f39558f" (UID: "343d315a-d25c-4d90-948b-fbc01f39558f"). InnerVolumeSpecName "kube-api-access-rzdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:30.821824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.821797 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" podUID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" containerName="payload-processing" containerID="cri-o://ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58" gracePeriod=25 Oct 29 22:47:30.846984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.846953 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" podUID="cfcd68a9-903b-4c02-83be-baf039321971" containerName="payload-pre-processing" containerID="cri-o://c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589" gracePeriod=25 Oct 29 22:47:30.893903 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.893878 4543 generic.go:356] "Generic (PLEG): container finished" podID="343d315a-d25c-4d90-948b-fbc01f39558f" containerID="71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2" exitCode=0 Oct 29 22:47:30.894006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.893931 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" Oct 29 22:47:30.894006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.893974 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" event={"ID":"343d315a-d25c-4d90-948b-fbc01f39558f","Type":"ContainerDied","Data":"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2"} Oct 29 22:47:30.894131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.894033 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v" event={"ID":"343d315a-d25c-4d90-948b-fbc01f39558f","Type":"ContainerDied","Data":"5e7fce859d71df99c5253540154ed3859af974031d38b404fa7525272f3bfa50"} Oct 29 22:47:30.894131 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.894061 4543 scope.go:122] "RemoveContainer" containerID="71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2" Oct 29 22:47:30.895399 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.895382 4543 generic.go:356] "Generic (PLEG): container finished" podID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" containerID="497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2" exitCode=0 Oct 29 22:47:30.895468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.895430 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" Oct 29 22:47:30.895534 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.895513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" event={"ID":"f9c09141-c0d9-4e29-a908-c6aedcc10f0e","Type":"ContainerDied","Data":"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2"} Oct 29 22:47:30.895606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.895565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs" event={"ID":"f9c09141-c0d9-4e29-a908-c6aedcc10f0e","Type":"ContainerDied","Data":"08ef331980a91250c33373f87e3411825661f5a645951b5fd3b5fa4d6a40671d"} Oct 29 22:47:30.900249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.900230 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\" (UniqueName: \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\") pod \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " Oct 29 22:47:30.900400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.900387 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\") pod \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\" (UID: \"f9c09141-c0d9-4e29-a908-c6aedcc10f0e\") " Oct 29 22:47:30.900636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.900621 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343d315a-d25c-4d90-948b-fbc01f39558f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:30.900701 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.900641 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdvz\" (UniqueName: \"kubernetes.io/projected/343d315a-d25c-4d90-948b-fbc01f39558f-kube-api-access-rzdvz\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:30.900752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.900707 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume" pod "f9c09141-c0d9-4e29-a908-c6aedcc10f0e" (UID: "f9c09141-c0d9-4e29-a908-c6aedcc10f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:30.902199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.902179 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv" pod "f9c09141-c0d9-4e29-a908-c6aedcc10f0e" (UID: "f9c09141-c0d9-4e29-a908-c6aedcc10f0e"). InnerVolumeSpecName "kube-api-access-s92zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:30.959067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.959046 4543 scope.go:122] "RemoveContainer" containerID="71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2" Oct 29 22:47:30.959390 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:30.959369 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2\": container with ID starting with 71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2 not found: ID does not exist" containerID="71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2" Oct 29 22:47:30.959460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.959397 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2"} err="failed to get container status \"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2\": rpc error: code = NotFound desc = could not find container \"71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2\": container with ID starting with 71a59baf7c40aef9fe15dc2777d41c8ecb92366da6a974100ef03573434f0ff2 not found: ID does not exist" Oct 29 22:47:30.959460 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.959414 4543 scope.go:122] "RemoveContainer" containerID="497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2" Oct 29 22:47:30.970069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.970053 4543 scope.go:122] "RemoveContainer" containerID="497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2" Oct 29 22:47:30.970420 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:30.970394 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2\": container with ID starting with 497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2 not found: ID does not exist" containerID="497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2" Oct 29 22:47:30.970521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.970427 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2"} err="failed to get container status \"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2\": rpc error: code = NotFound desc = could not find container \"497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2\": container with ID starting with 497045da0b1ec6ac3adb31c29daa06f4ffe802e4ab7efd5fc70dcaf518c1b2b2 not found: ID does not exist" Oct 29 22:47:30.991606 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.991574 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 22:47:30.997469 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:30.997431 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-863f1b-57d45b8958-bqx7v"] Oct 29 22:47:31.001886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.001858 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s92zv\" (UniqueName: \"kubernetes.io/projected/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-kube-api-access-s92zv\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.001886 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.001884 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c09141-c0d9-4e29-a908-c6aedcc10f0e-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.035589 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.035550 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj"] Oct 29 22:47:31.039855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.039828 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cd7c7c-h6gpj"] Oct 29 22:47:31.093040 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.093015 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:47:31.125046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.125023 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:47:31.204314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.204255 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\") pod \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " Oct 29 22:47:31.204314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.204297 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\" (UniqueName: \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\") pod \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\" (UID: \"57d5410e-d4e5-4e4f-b399-4d42ab459d02\") " Oct 29 22:47:31.204707 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.204675 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume" pod "57d5410e-d4e5-4e4f-b399-4d42ab459d02" (UID: "57d5410e-d4e5-4e4f-b399-4d42ab459d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:31.204801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.204751 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57d5410e-d4e5-4e4f-b399-4d42ab459d02-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.206450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.206426 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm"] Oct 29 22:47:31.207067 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207048 4543 cpu_manager.go:414] "containerMap: removing container" podUID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" containerName="payload-processing" Oct 29 22:47:31.207158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207093 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" containerName="payload-processing" Oct 29 22:47:31.207158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207114 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cfcd68a9-903b-4c02-83be-baf039321971" containerName="payload-pre-processing" Oct 29 22:47:31.207158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207122 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcd68a9-903b-4c02-83be-baf039321971" containerName="payload-pre-processing" Oct 29 22:47:31.207158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207145 4543 cpu_manager.go:414] "containerMap: removing container" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" Oct 29 22:47:31.207158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207153 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207173 4543 cpu_manager.go:414] "containerMap: removing container" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207180 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207201 4543 cpu_manager.go:414] "containerMap: removing container" podUID="343d315a-d25c-4d90-948b-fbc01f39558f" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207208 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343d315a-d25c-4d90-948b-fbc01f39558f" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207217 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207224 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207242 4543 cpu_manager.go:414] "containerMap: removing container" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207249 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207281 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d65663dd-47a0-4a7c-bf68-89672a384efa" containerName="revoke-keys" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207288 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65663dd-47a0-4a7c-bf68-89672a384efa" containerName="revoke-keys" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207301 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" containerName="payload-pre-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207309 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" containerName="payload-pre-processing" Oct 29 22:47:31.207319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207323 4543 cpu_manager.go:414] "containerMap: removing container" podUID="784c993f-9ccd-4909-b4fd-2473476582fb" containerName="revoke-keys" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207330 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784c993f-9ccd-4909-b4fd-2473476582fb" containerName="revoke-keys" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207439 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="718bbebf-8ef8-4311-a934-7b28acfc5cfd" containerName="payload-pre-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207455 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="983248db-dedd-4c0e-9750-e4fc85e79e0b" containerName="payload-pre-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207469 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="343d315a-d25c-4d90-948b-fbc01f39558f" containerName="payload-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207478 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5924a540-1317-4e5a-9251-9ced453c35be" containerName="payload-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207488 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="937e2d8e-bab2-4515-bcfb-2dcc5c6c5a9b" containerName="payload-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207501 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65663dd-47a0-4a7c-bf68-89672a384efa" containerName="revoke-keys" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207510 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" containerName="payload-pre-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207520 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="784c993f-9ccd-4909-b4fd-2473476582fb" containerName="revoke-keys" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207448 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr" pod "57d5410e-d4e5-4e4f-b399-4d42ab459d02" (UID: "57d5410e-d4e5-4e4f-b399-4d42ab459d02"). InnerVolumeSpecName "kube-api-access-r4ttr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207530 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" containerName="payload-processing" Oct 29 22:47:31.207960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.207573 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcd68a9-903b-4c02-83be-baf039321971" containerName="payload-pre-processing" Oct 29 22:47:31.211283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.211262 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.227062 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.226808 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp"] Oct 29 22:47:31.229950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.229930 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:47:31.231201 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.231183 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm"] Oct 29 22:47:31.231282 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.231262 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.232194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.232157 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:47:31.232998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.232978 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:47:31.233119 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.233010 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:47:31.248435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.248413 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 22:47:31.251418 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.251340 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp"] Oct 29 22:47:31.259699 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.259648 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-863f1b-dd8bf8bbc-28gbs"] Oct 29 22:47:31.306028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.305983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\") pod \"cfcd68a9-903b-4c02-83be-baf039321971\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " Oct 29 22:47:31.306269 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.306103 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\" (UniqueName: \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\") pod \"cfcd68a9-903b-4c02-83be-baf039321971\" (UID: \"cfcd68a9-903b-4c02-83be-baf039321971\") " Oct 29 22:47:31.306535 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.306509 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rmn\" (UniqueName: \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.306630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.306585 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.306630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.306611 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.306743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.306730 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ttr\" (UniqueName: \"kubernetes.io/projected/57d5410e-d4e5-4e4f-b399-4d42ab459d02-kube-api-access-r4ttr\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.307190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.307159 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume" pod "cfcd68a9-903b-4c02-83be-baf039321971" (UID: "cfcd68a9-903b-4c02-83be-baf039321971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:31.311235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.311203 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6" pod "cfcd68a9-903b-4c02-83be-baf039321971" (UID: "cfcd68a9-903b-4c02-83be-baf039321971"). InnerVolumeSpecName "kube-api-access-6lcc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:31.407504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407471 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdkf\" (UniqueName: \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.407504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407504 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rmn\" (UniqueName: \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407715 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407734 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407859 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcd68a9-903b-4c02-83be-baf039321971-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.407908 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.407873 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lcc6\" (UniqueName: \"kubernetes.io/projected/cfcd68a9-903b-4c02-83be-baf039321971-kube-api-access-6lcc6\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.408491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.408468 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.410128 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.410110 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.434846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.434822 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rmn\" (UniqueName: \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\") pod \"maas-api-revoke-sub-7cab32cf-28pgm\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.509019 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.508998 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.509142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.509048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdkf\" (UniqueName: \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.509142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.509067 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.509630 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.509608 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.511240 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.511221 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.529743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.529723 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:31.541639 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.541611 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdkf\" (UniqueName: \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\") pod \"maas-api-revoke-sub-69764cd9-8knrp\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.554285 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.554257 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:31.873141 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.873116 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm"] Oct 29 22:47:31.875057 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:47:31.875022 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbb372f_c0cf_4df3_b895_a121136542f2.slice/crio-552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7 WatchSource:0}: Error finding container 552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7: Status 404 returned error can't find the container with id 552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7 Oct 29 22:47:31.903962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.903932 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" event={"ID":"3dbb372f-c0cf-4df3-b895-a121136542f2","Type":"ContainerStarted","Data":"552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7"} Oct 29 22:47:31.905249 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.905226 4543 generic.go:356] "Generic (PLEG): container finished" podID="cfcd68a9-903b-4c02-83be-baf039321971" containerID="c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589" exitCode=0 Oct 29 22:47:31.905359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.905293 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" event={"ID":"cfcd68a9-903b-4c02-83be-baf039321971","Type":"ContainerDied","Data":"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589"} Oct 29 22:47:31.905359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.905314 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" Oct 29 22:47:31.905359 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.905348 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc" event={"ID":"cfcd68a9-903b-4c02-83be-baf039321971","Type":"ContainerDied","Data":"ae490805a41b9d18afddfd084123fafb4399b21997f984f2d8e06d930bf35f6f"} Oct 29 22:47:31.905557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.905378 4543 scope.go:122] "RemoveContainer" containerID="c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589" Oct 29 22:47:31.907054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.907031 4543 generic.go:356] "Generic (PLEG): container finished" podID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" containerID="ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58" exitCode=0 Oct 29 22:47:31.907982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.907475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" event={"ID":"57d5410e-d4e5-4e4f-b399-4d42ab459d02","Type":"ContainerDied","Data":"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58"} Oct 29 22:47:31.907982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.907522 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" event={"ID":"57d5410e-d4e5-4e4f-b399-4d42ab459d02","Type":"ContainerDied","Data":"b2f592f3da9fa9e724243cd27eb7b9779f9ed1e4882afe81b681b7ee760f08c9"} Oct 29 22:47:31.907982 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.907823 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b" Oct 29 22:47:31.913347 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.911965 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp"] Oct 29 22:47:31.918625 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.918610 4543 scope.go:122] "RemoveContainer" containerID="c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589" Oct 29 22:47:31.918875 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:31.918850 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589\": container with ID starting with c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589 not found: ID does not exist" containerID="c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589" Oct 29 22:47:31.918988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.918881 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589"} err="failed to get container status \"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589\": rpc error: code = NotFound desc = could not find container \"c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589\": container with ID starting with c3b59ffb5e94072c16a300ca1a12bdd9d5921decdaa4f46170c280f000753589 not found: ID does not exist" Oct 29 22:47:31.918988 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.918904 4543 scope.go:122] "RemoveContainer" containerID="ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58" Oct 29 22:47:31.930238 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.930216 4543 scope.go:122] "RemoveContainer" containerID="ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58" Oct 29 22:47:31.930513 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:31.930494 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58\": container with ID starting with ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58 not found: ID does not exist" containerID="ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58" Oct 29 22:47:31.930564 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.930520 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58"} err="failed to get container status \"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58\": rpc error: code = NotFound desc = could not find container \"ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58\": container with ID starting with ca39aa9a7f13abbf475dbb056622f6a37a68f29871e6df0eabd65606745e3f58 not found: ID does not exist" Oct 29 22:47:31.966420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.966395 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:47:31.991539 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:31.991511 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-201a3d-7db959c57b-ggt9b"] Oct 29 22:47:32.015781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.015760 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:47:32.023623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.023605 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-201a3d-5b95967999-skmzc"] Oct 29 22:47:32.384744 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.384712 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343d315a-d25c-4d90-948b-fbc01f39558f" path="/var/lib/kubelet/pods/343d315a-d25c-4d90-948b-fbc01f39558f/volumes" Oct 29 22:47:32.385137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.385122 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d5410e-d4e5-4e4f-b399-4d42ab459d02" path="/var/lib/kubelet/pods/57d5410e-d4e5-4e4f-b399-4d42ab459d02/volumes" Oct 29 22:47:32.385504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.385491 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21e98d5-42fd-4e1e-885e-0050da2409c4" path="/var/lib/kubelet/pods/b21e98d5-42fd-4e1e-885e-0050da2409c4/volumes" Oct 29 22:47:32.386139 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.386119 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcd68a9-903b-4c02-83be-baf039321971" path="/var/lib/kubelet/pods/cfcd68a9-903b-4c02-83be-baf039321971/volumes" Oct 29 22:47:32.386507 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.386494 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c09141-c0d9-4e29-a908-c6aedcc10f0e" path="/var/lib/kubelet/pods/f9c09141-c0d9-4e29-a908-c6aedcc10f0e/volumes" Oct 29 22:47:32.927013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.926918 4543 generic.go:356] "Generic (PLEG): container finished" podID="3dbb372f-c0cf-4df3-b895-a121136542f2" containerID="144b33d7a7922a79f995cbba4beb3ed86ab65c5a455bfa9b6a655bfecce8b2fe" exitCode=0 Oct 29 22:47:32.927013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.927002 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" event={"ID":"3dbb372f-c0cf-4df3-b895-a121136542f2","Type":"ContainerDied","Data":"144b33d7a7922a79f995cbba4beb3ed86ab65c5a455bfa9b6a655bfecce8b2fe"} Oct 29 22:47:32.929491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.929467 4543 generic.go:356] "Generic (PLEG): container finished" podID="2c8f8750-de5c-4795-9c7e-97ebda53b478" containerID="6019a91c6388c23b1e92d20f96e1fa0c385eacf1608811229c9a445eea9b36dc" exitCode=0 Oct 29 22:47:32.929648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.929533 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" event={"ID":"2c8f8750-de5c-4795-9c7e-97ebda53b478","Type":"ContainerDied","Data":"6019a91c6388c23b1e92d20f96e1fa0c385eacf1608811229c9a445eea9b36dc"} Oct 29 22:47:32.929648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:32.929577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" event={"ID":"2c8f8750-de5c-4795-9c7e-97ebda53b478","Type":"ContainerStarted","Data":"ef920510699b2a374c0c3dbc5cd91c34b7fd539413ba762c11d7ddbc23aac4ad"} Oct 29 22:47:34.076487 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.076456 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:34.106520 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.106495 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:34.245137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245102 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\" (UniqueName: \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\") pod \"3dbb372f-c0cf-4df3-b895-a121136542f2\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " Oct 29 22:47:34.245279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245209 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\") pod \"2c8f8750-de5c-4795-9c7e-97ebda53b478\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " Oct 29 22:47:34.245279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245240 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\") pod \"3dbb372f-c0cf-4df3-b895-a121136542f2\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " Oct 29 22:47:34.245428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245289 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\" (UniqueName: \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\") pod \"2c8f8750-de5c-4795-9c7e-97ebda53b478\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " Oct 29 22:47:34.245428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245314 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\") pod \"2c8f8750-de5c-4795-9c7e-97ebda53b478\" (UID: \"2c8f8750-de5c-4795-9c7e-97ebda53b478\") " Oct 29 22:47:34.245428 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245358 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\") pod \"3dbb372f-c0cf-4df3-b895-a121136542f2\" (UID: \"3dbb372f-c0cf-4df3-b895-a121136542f2\") " Oct 29 22:47:34.245640 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245612 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca" pod "2c8f8750-de5c-4795-9c7e-97ebda53b478" (UID: "2c8f8750-de5c-4795-9c7e-97ebda53b478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:34.245818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.245795 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca" pod "3dbb372f-c0cf-4df3-b895-a121136542f2" (UID: "3dbb372f-c0cf-4df3-b895-a121136542f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:34.247549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.247523 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf" pod "2c8f8750-de5c-4795-9c7e-97ebda53b478" (UID: "2c8f8750-de5c-4795-9c7e-97ebda53b478"). InnerVolumeSpecName "kube-api-access-ttdkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:34.247549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.247540 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca" pod "3dbb372f-c0cf-4df3-b895-a121136542f2" (UID: "3dbb372f-c0cf-4df3-b895-a121136542f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:34.247932 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.247910 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca" pod "2c8f8750-de5c-4795-9c7e-97ebda53b478" (UID: "2c8f8750-de5c-4795-9c7e-97ebda53b478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:34.247994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.247929 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn" pod "3dbb372f-c0cf-4df3-b895-a121136542f2" (UID: "3dbb372f-c0cf-4df3-b895-a121136542f2"). InnerVolumeSpecName "kube-api-access-z2rmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:34.346838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346804 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdkf\" (UniqueName: \"kubernetes.io/projected/2c8f8750-de5c-4795-9c7e-97ebda53b478-kube-api-access-ttdkf\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.346838 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346837 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.347012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346848 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.347012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346857 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rmn\" (UniqueName: \"kubernetes.io/projected/3dbb372f-c0cf-4df3-b895-a121136542f2-kube-api-access-z2rmn\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.347012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346865 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8f8750-de5c-4795-9c7e-97ebda53b478-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.347012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.346873 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbb372f-c0cf-4df3-b895-a121136542f2-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:34.617329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.617238 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:47:34.617548 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.617519 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" containerID="cri-o://795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29" gracePeriod=30 Oct 29 22:47:34.945628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.945547 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" Oct 29 22:47:34.945794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.945552 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cab32cf-28pgm" event={"ID":"3dbb372f-c0cf-4df3-b895-a121136542f2","Type":"ContainerDied","Data":"552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7"} Oct 29 22:47:34.945794 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.945693 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552491fa3db11b69d0944a452280e694bf43a93afc1874b382818561b93131c7" Oct 29 22:47:34.947176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.947146 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" Oct 29 22:47:34.947343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.947177 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69764cd9-8knrp" event={"ID":"2c8f8750-de5c-4795-9c7e-97ebda53b478","Type":"ContainerDied","Data":"ef920510699b2a374c0c3dbc5cd91c34b7fd539413ba762c11d7ddbc23aac4ad"} Oct 29 22:47:34.947343 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.947213 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef920510699b2a374c0c3dbc5cd91c34b7fd539413ba762c11d7ddbc23aac4ad" Oct 29 22:47:34.989536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:34.989512 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" containerID="cri-o://925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7" gracePeriod=45 Oct 29 22:47:35.238432 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.238407 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:47:35.254962 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.254936 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2"] Oct 29 22:47:35.255423 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255407 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" Oct 29 22:47:35.255468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255425 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" Oct 29 22:47:35.255468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255437 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2c8f8750-de5c-4795-9c7e-97ebda53b478" containerName="revoke-keys" Oct 29 22:47:35.255468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255442 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8f8750-de5c-4795-9c7e-97ebda53b478" containerName="revoke-keys" Oct 29 22:47:35.255468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255462 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3dbb372f-c0cf-4df3-b895-a121136542f2" containerName="revoke-keys" Oct 29 22:47:35.255468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255468 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbb372f-c0cf-4df3-b895-a121136542f2" containerName="revoke-keys" Oct 29 22:47:35.255637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255534 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerName="main" Oct 29 22:47:35.255637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255542 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8f8750-de5c-4795-9c7e-97ebda53b478" containerName="revoke-keys" Oct 29 22:47:35.255637 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.255552 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbb372f-c0cf-4df3-b895-a121136542f2" containerName="revoke-keys" Oct 29 22:47:35.258547 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.258531 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.276193 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.276170 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:47:35.276333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.276248 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:47:35.279013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.278981 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2"] Oct 29 22:47:35.302880 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.302857 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:47:35.303029 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.303015 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:47:35.356069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356037 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\" (UniqueName: \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.356289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356269 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.356416 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356401 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.356493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356473 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.356562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356518 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.356562 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356550 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\") pod \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\" (UID: \"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06\") " Oct 29 22:47:35.357028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356684 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.357028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356809 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.357028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356812 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.357028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356871 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.357028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.356901 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xsk\" (UniqueName: \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.357587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.357128 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.357587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.357247 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.357587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.357265 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.357587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.357279 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.360818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.360790 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:35.360945 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.360847 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.361268 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.361237 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp" pod "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" (UID: "e5c8048b-cb51-4f60-81b2-ed9dab8f8a06"). InnerVolumeSpecName "kube-api-access-vg5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:35.457812 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.457781 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xsk\" (UniqueName: \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.457994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.457897 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.458066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.458000 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.458169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.458066 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5qp\" (UniqueName: \"kubernetes.io/projected/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-kube-api-access-vg5qp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.458169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.458113 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.458169 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.458129 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.458676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.458648 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.460598 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.460579 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.483626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.483564 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xsk\" (UniqueName: \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.571385 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.571364 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:35.878464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.878440 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2"] Oct 29 22:47:35.880643 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:47:35.880614 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57070eb3_a030_4897_99ff_d57a2d246d06.slice/crio-419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3 WatchSource:0}: Error finding container 419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3: Status 404 returned error can't find the container with id 419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3 Oct 29 22:47:35.955220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.955199 4543 generic.go:356] "Generic (PLEG): container finished" podID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" containerID="925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7" exitCode=0 Oct 29 22:47:35.955335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.955243 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" Oct 29 22:47:35.955335 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.955290 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" event={"ID":"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06","Type":"ContainerDied","Data":"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7"} Oct 29 22:47:35.955479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.955338 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl" event={"ID":"e5c8048b-cb51-4f60-81b2-ed9dab8f8a06","Type":"ContainerDied","Data":"9953f19fcec512e322aa25d3c9eb4b0fab9910b27ad828dcb6ae8dd4c21c55e4"} Oct 29 22:47:35.955479 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.955364 4543 scope.go:122] "RemoveContainer" containerID="925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7" Oct 29 22:47:35.956692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.956654 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" event={"ID":"57070eb3-a030-4897-99ff-d57a2d246d06","Type":"ContainerStarted","Data":"419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3"} Oct 29 22:47:35.964881 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.964864 4543 scope.go:122] "RemoveContainer" containerID="925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7" Oct 29 22:47:35.965167 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:35.965146 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7\": container with ID starting with 925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7 not found: ID does not exist" containerID="925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7" Oct 29 22:47:35.965233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:35.965176 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7"} err="failed to get container status \"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7\": rpc error: code = NotFound desc = could not find container \"925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7\": container with ID starting with 925664681c91f0c449a0f6a0ce6fc2917b059de568fbb65896dd2f2c2b0076a7 not found: ID does not exist" Oct 29 22:47:36.013814 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.013794 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 22:47:36.020388 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.020368 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/distinct-2-w6-863f1b-kserve-84b8fcff76-9t9pl"] Oct 29 22:47:36.325125 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.325069 4543 patch_prober.go:29] interesting pod/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:47:36.325430 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.325149 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:47:36.385517 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.385488 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c8048b-cb51-4f60-81b2-ed9dab8f8a06" path="/var/lib/kubelet/pods/e5c8048b-cb51-4f60-81b2-ed9dab8f8a06/volumes" Oct 29 22:47:36.965743 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.965708 4543 generic.go:356] "Generic (PLEG): container finished" podID="57070eb3-a030-4897-99ff-d57a2d246d06" containerID="a6747f7550d8099667fa53912efd05d81b13664a66b7fa53fbc3a7869ccb84e3" exitCode=0 Oct 29 22:47:36.965920 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:36.965747 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" event={"ID":"57070eb3-a030-4897-99ff-d57a2d246d06","Type":"ContainerDied","Data":"a6747f7550d8099667fa53912efd05d81b13664a66b7fa53fbc3a7869ccb84e3"} Oct 29 22:47:38.035969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.035930 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk"] Oct 29 22:47:38.048576 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.048547 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064d209c-btrkk"] Oct 29 22:47:38.105028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.105003 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:38.285616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.285540 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\") pod \"57070eb3-a030-4897-99ff-d57a2d246d06\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " Oct 29 22:47:38.285616 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.285612 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\") pod \"57070eb3-a030-4897-99ff-d57a2d246d06\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " Oct 29 22:47:38.285798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.285767 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\" (UniqueName: \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\") pod \"57070eb3-a030-4897-99ff-d57a2d246d06\" (UID: \"57070eb3-a030-4897-99ff-d57a2d246d06\") " Oct 29 22:47:38.285964 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.285927 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca" pod "57070eb3-a030-4897-99ff-d57a2d246d06" (UID: "57070eb3-a030-4897-99ff-d57a2d246d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:38.286132 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.286114 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:38.287624 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.287604 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca" pod "57070eb3-a030-4897-99ff-d57a2d246d06" (UID: "57070eb3-a030-4897-99ff-d57a2d246d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:38.287865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.287846 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk" pod "57070eb3-a030-4897-99ff-d57a2d246d06" (UID: "57070eb3-a030-4897-99ff-d57a2d246d06"). InnerVolumeSpecName "kube-api-access-g2xsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:38.386676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.386647 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57070eb3-a030-4897-99ff-d57a2d246d06-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:38.386676 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.386677 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xsk\" (UniqueName: \"kubernetes.io/projected/57070eb3-a030-4897-99ff-d57a2d246d06-kube-api-access-g2xsk\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:38.386861 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.386646 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3" path="/var/lib/kubelet/pods/1d3618d3-1f3e-483e-bfbf-6c1b1a3072e3/volumes" Oct 29 22:47:38.980670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.980631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" event={"ID":"57070eb3-a030-4897-99ff-d57a2d246d06","Type":"ContainerDied","Data":"419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3"} Oct 29 22:47:38.980855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.980675 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2" Oct 29 22:47:38.980855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:38.980690 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="419e8fe8a51d7b8c601bf9bf402a3b382b81fcefefc92025a04693326d634ff3" Oct 29 22:47:39.045865 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.045824 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv"] Oct 29 22:47:39.058211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.058170 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b2e232-kbftv"] Oct 29 22:47:39.887022 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.886967 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:47:39.916130 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916098 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916134 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916233 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916189 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916259 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916279 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916319 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916314 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916353 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916387 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916427 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\") pod \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\" (UID: \"e6b613ca-f1c4-4764-b23d-f98afd462fa4\") " Oct 29 22:47:39.916531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916453 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:39.916531 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916478 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:39.916809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:39.916809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:39.916809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916783 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:39.916809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916801 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:39.917055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916818 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:39.917055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.916835 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:39.917055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.917007 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:39.918683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.918659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:39.918683 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.918669 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:39.918835 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.918706 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "kube-api-access-w798f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:39.919027 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.919004 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token" pod "e6b613ca-f1c4-4764-b23d-f98afd462fa4" (UID: "e6b613ca-f1c4-4764-b23d-f98afd462fa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:39.988521 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.988495 4543 generic.go:356] "Generic (PLEG): container finished" podID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerID="795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29" exitCode=0 Oct 29 22:47:39.988666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.988546 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" Oct 29 22:47:39.988666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.988589 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerDied","Data":"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29"} Oct 29 22:47:39.988666 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.988662 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8" event={"ID":"e6b613ca-f1c4-4764-b23d-f98afd462fa4","Type":"ContainerDied","Data":"934512e213055df533863262a040d902b6df69fa7014a5bef886c22a03a8f6b3"} Oct 29 22:47:39.988803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:39.988689 4543 scope.go:122] "RemoveContainer" containerID="795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29" Oct 29 22:47:40.002904 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.002886 4543 scope.go:122] "RemoveContainer" containerID="a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5" Oct 29 22:47:40.011883 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.011866 4543 scope.go:122] "RemoveContainer" containerID="795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29" Oct 29 22:47:40.012169 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:40.012149 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29\": container with ID starting with 795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29 not found: ID does not exist" containerID="795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29" Oct 29 22:47:40.012221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.012178 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29"} err="failed to get container status \"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29\": rpc error: code = NotFound desc = could not find container \"795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29\": container with ID starting with 795f894c33d46f49868a10603dace19e3171b9953254e3975817006204446d29 not found: ID does not exist" Oct 29 22:47:40.012221 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.012193 4543 scope.go:122] "RemoveContainer" containerID="a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5" Oct 29 22:47:40.012459 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:40.012440 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5\": container with ID starting with a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5 not found: ID does not exist" containerID="a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5" Oct 29 22:47:40.012522 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.012469 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5"} err="failed to get container status \"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5\": rpc error: code = NotFound desc = could not find container \"a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5\": container with ID starting with a0b4685ca9fb6320aac256b9328cb702e4cbeab54afad4fa859c056aa68002a5 not found: ID does not exist" Oct 29 22:47:40.018017 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.017998 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:40.018149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.018021 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w798f\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-kube-api-access-w798f\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:40.018149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.018036 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:40.018149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.018049 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b613ca-f1c4-4764-b23d-f98afd462fa4-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:40.018149 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.018060 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b613ca-f1c4-4764-b23d-f98afd462fa4-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:47:40.049782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.049755 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:47:40.055981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.055957 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-201a3d-openshift-default-7b877d8dfc-fphw8"] Oct 29 22:47:40.384464 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.384427 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14367e8f-e349-46dc-9535-2822050b29e0" path="/var/lib/kubelet/pods/14367e8f-e349-46dc-9535-2822050b29e0/volumes" Oct 29 22:47:40.384888 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:40.384873 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" path="/var/lib/kubelet/pods/e6b613ca-f1c4-4764-b23d-f98afd462fa4/volumes" Oct 29 22:47:42.046800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.046761 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx"] Oct 29 22:47:42.060055 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.060024 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb"] Oct 29 22:47:42.062232 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.062217 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c005b17-ddwvx"] Oct 29 22:47:42.064054 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.064033 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee52575a-sbzgb"] Oct 29 22:47:42.384790 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.384725 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fd0b54-6b3c-4a29-b4ff-95db857b425f" path="/var/lib/kubelet/pods/27fd0b54-6b3c-4a29-b4ff-95db857b425f/volumes" Oct 29 22:47:42.385170 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:42.385155 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d" path="/var/lib/kubelet/pods/aece6e8b-90f4-420b-a3cb-ead1ffaf3b3d/volumes" Oct 29 22:47:45.763279 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:45.763240 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:47:45.803604 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:45.803573 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:47:50.443636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:50.443602 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc"] Oct 29 22:47:50.456832 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:50.456809 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-863f1b-pp2kc"] Oct 29 22:47:50.764186 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:50.764147 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" containerID="cri-o://6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3" gracePeriod=25 Oct 29 22:47:50.803916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:50.803877 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" containerID="cri-o://3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831" gracePeriod=25 Oct 29 22:47:52.383806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:52.383771 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784c993f-9ccd-4909-b4fd-2473476582fb" path="/var/lib/kubelet/pods/784c993f-9ccd-4909-b4fd-2473476582fb/volumes" Oct 29 22:47:52.423653 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:52.423628 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2"] Oct 29 22:47:52.428658 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:52.428634 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-1e74c3-5nhp2"] Oct 29 22:47:54.383819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:54.383784 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57070eb3-a030-4897-99ff-d57a2d246d06" path="/var/lib/kubelet/pods/57070eb3-a030-4897-99ff-d57a2d246d06/volumes" Oct 29 22:47:57.960962 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:47:57.960928 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:47:58.335524 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:58.335446 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:47:58.335714 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:58.335669 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" containerID="cri-o://cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59" gracePeriod=30 Oct 29 22:47:58.342293 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:58.342253 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:47:58.342435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:47:58.342308 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:02.216142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216105 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc"] Oct 29 22:48:02.216700 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216679 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:02.216806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216703 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:02.216806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216738 4543 cpu_manager.go:414] "containerMap: removing container" podUID="57070eb3-a030-4897-99ff-d57a2d246d06" containerName="revoke-keys" Oct 29 22:48:02.216806 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216746 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57070eb3-a030-4897-99ff-d57a2d246d06" containerName="revoke-keys" Oct 29 22:48:02.216986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216900 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:02.216986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216927 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:02.216986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.216940 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="57070eb3-a030-4897-99ff-d57a2d246d06" containerName="revoke-keys" Oct 29 22:48:02.220326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.220306 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.244443 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.244416 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc"] Oct 29 22:48:02.255158 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.255139 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:48:02.255263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.255187 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:48:02.291991 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.291972 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:02.292401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.292384 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:48:02.320045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.320020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.320154 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.320065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.320202 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.320161 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thshh\" (UniqueName: \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.421455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.421433 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.421545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.421473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.421545 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.421520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thshh\" (UniqueName: \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.422069 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.422049 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.423978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.423957 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.449565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.449543 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thshh\" (UniqueName: \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.532896 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.532846 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:02.838879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:02.838796 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc"] Oct 29 22:48:02.841634 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:48:02.841606 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccf6fc8_ddc5_4fcc_831e_38b6b7aec316.slice/crio-e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7 WatchSource:0}: Error finding container e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7: Status 404 returned error can't find the container with id e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7 Oct 29 22:48:03.142394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.142361 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" event={"ID":"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316","Type":"ContainerStarted","Data":"e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7"} Oct 29 22:48:03.592298 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.592275 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:48:03.735278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735249 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735299 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735333 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735352 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735407 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735381 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735529 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.735648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735648 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735625 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735688 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.735818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735705 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735767 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\") pod \"bc5097e4-78c1-4d74-9759-d5198b887368\" (UID: \"bc5097e4-78c1-4d74-9759-d5198b887368\") " Oct 29 22:48:03.735818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735702 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.735818 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735772 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:03.736289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.735867 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.736893 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.736856 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.737037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.736901 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.737037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.736923 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5097e4-78c1-4d74-9759-d5198b887368-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.737037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.736938 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.737037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.736969 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.738326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.738300 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.745859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.745817 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:03.750046 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.750013 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "kube-api-access-dhjnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:03.750556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.750533 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo" pod "bc5097e4-78c1-4d74-9759-d5198b887368" (UID: "bc5097e4-78c1-4d74-9759-d5198b887368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:03.837942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.837923 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.837942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.837945 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjnq\" (UniqueName: \"kubernetes.io/projected/bc5097e4-78c1-4d74-9759-d5198b887368-kube-api-access-dhjnq\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.838108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.837954 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5097e4-78c1-4d74-9759-d5198b887368-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.838108 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:03.837963 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5097e4-78c1-4d74-9759-d5198b887368-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:04.150088 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.150015 4543 generic.go:356] "Generic (PLEG): container finished" podID="eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" containerID="d7fa3abcbc8c4afccc40a4333391e1626896b58aa0f8bb2ba48e4f5c35fdeb12" exitCode=0 Oct 29 22:48:04.150194 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.150103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" event={"ID":"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316","Type":"ContainerDied","Data":"d7fa3abcbc8c4afccc40a4333391e1626896b58aa0f8bb2ba48e4f5c35fdeb12"} Oct 29 22:48:04.151557 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.151536 4543 generic.go:356] "Generic (PLEG): container finished" podID="bc5097e4-78c1-4d74-9759-d5198b887368" containerID="cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59" exitCode=0 Oct 29 22:48:04.151661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.151590 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" Oct 29 22:48:04.151661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.151622 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" event={"ID":"bc5097e4-78c1-4d74-9759-d5198b887368","Type":"ContainerDied","Data":"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59"} Oct 29 22:48:04.151758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.151673 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf" event={"ID":"bc5097e4-78c1-4d74-9759-d5198b887368","Type":"ContainerDied","Data":"2fb66ec433658605ffca0e56798565318f1e2baaab8d2150a9840467e1d3d9bc"} Oct 29 22:48:04.151758 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.151698 4543 scope.go:122] "RemoveContainer" containerID="cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59" Oct 29 22:48:04.162339 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.162322 4543 scope.go:122] "RemoveContainer" containerID="cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59" Oct 29 22:48:04.162583 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:04.162564 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59\": container with ID starting with cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59 not found: ID does not exist" containerID="cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59" Oct 29 22:48:04.162636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.162592 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59"} err="failed to get container status \"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59\": rpc error: code = NotFound desc = could not find container \"cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59\": container with ID starting with cb14ef34da3c484df7538a2d1b35ad71657dcb0bc2dc5c8f1ff6b3e634ac3e59 not found: ID does not exist" Oct 29 22:48:04.235137 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.235118 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:48:04.247960 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.247933 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-7xjxf"] Oct 29 22:48:04.384556 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:04.384528 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" path="/var/lib/kubelet/pods/bc5097e4-78c1-4d74-9759-d5198b887368/volumes" Oct 29 22:48:05.307213 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.307189 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:05.453007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.452946 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\") pod \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " Oct 29 22:48:05.453162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.453069 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\" (UniqueName: \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\") pod \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " Oct 29 22:48:05.453162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.453117 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\") pod \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\" (UID: \"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316\") " Oct 29 22:48:05.453544 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.453512 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca" pod "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" (UID: "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:05.455162 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.455139 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca" pod "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" (UID: "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:05.455394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.455379 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh" pod "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" (UID: "eccf6fc8-ddc5-4fcc-831e-38b6b7aec316"). InnerVolumeSpecName "kube-api-access-thshh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:05.554133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.554107 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thshh\" (UniqueName: \"kubernetes.io/projected/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-kube-api-access-thshh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:05.554133 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.554129 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:05.554283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:05.554138 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:06.172109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:06.172055 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" event={"ID":"eccf6fc8-ddc5-4fcc-831e-38b6b7aec316","Type":"ContainerDied","Data":"e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7"} Oct 29 22:48:06.172109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:06.172103 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc" Oct 29 22:48:06.172322 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:06.172125 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60889c6aa4a660a829e59f09827f06cb81367eed6f21712ff17a3d5928027c7" Oct 29 22:48:08.342673 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:08.342626 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:08.343106 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:08.342638 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:11.056659 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.056633 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:48:11.088614 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.088587 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:48:11.205059 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.205026 4543 generic.go:356] "Generic (PLEG): container finished" podID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerID="3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831" exitCode=0 Oct 29 22:48:11.205263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.205110 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" Oct 29 22:48:11.205263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.205113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" event={"ID":"393e6e93-fcde-4371-adc3-b0a888f3041f","Type":"ContainerDied","Data":"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831"} Oct 29 22:48:11.205263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.205211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv" event={"ID":"393e6e93-fcde-4371-adc3-b0a888f3041f","Type":"ContainerDied","Data":"37ba81d2e77dbaefc68f1854a2985564a617a9ee31b826513b4418da3e8dd50f"} Oct 29 22:48:11.205263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.205247 4543 scope.go:122] "RemoveContainer" containerID="3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831" Oct 29 22:48:11.206294 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206257 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\") pod \"393e6e93-fcde-4371-adc3-b0a888f3041f\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " Oct 29 22:48:11.206378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206321 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\") pod \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " Oct 29 22:48:11.206378 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206350 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\" (UniqueName: \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\") pod \"393e6e93-fcde-4371-adc3-b0a888f3041f\" (UID: \"393e6e93-fcde-4371-adc3-b0a888f3041f\") " Oct 29 22:48:11.206491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206405 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\" (UniqueName: \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\") pod \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\" (UID: \"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8\") " Oct 29 22:48:11.206591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206568 4543 generic.go:356] "Generic (PLEG): container finished" podID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerID="6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3" exitCode=0 Oct 29 22:48:11.206778 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206745 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume" pod "d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" (UID: "d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:11.206910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206786 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume" pod "393e6e93-fcde-4371-adc3-b0a888f3041f" (UID: "393e6e93-fcde-4371-adc3-b0a888f3041f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:11.206910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206795 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" event={"ID":"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8","Type":"ContainerDied","Data":"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3"} Oct 29 22:48:11.206910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206836 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" event={"ID":"d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8","Type":"ContainerDied","Data":"ea420e3ee123df33667bb9714f7bb2512c0d7cded2e06b9b1cebc78f57fa2bc1"} Oct 29 22:48:11.206910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206844 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd" Oct 29 22:48:11.206910 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.206863 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.208698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.208671 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r" pod "d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" (UID: "d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8"). InnerVolumeSpecName "kube-api-access-mbv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:11.208809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.208791 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9" pod "393e6e93-fcde-4371-adc3-b0a888f3041f" (UID: "393e6e93-fcde-4371-adc3-b0a888f3041f"). InnerVolumeSpecName "kube-api-access-vpbp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:11.215397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.215380 4543 scope.go:122] "RemoveContainer" containerID="3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831" Oct 29 22:48:11.215669 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:11.215650 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831\": container with ID starting with 3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831 not found: ID does not exist" containerID="3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831" Oct 29 22:48:11.215751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.215680 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831"} err="failed to get container status \"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831\": rpc error: code = NotFound desc = could not find container \"3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831\": container with ID starting with 3d5e3b3ec427cc1ae204e701fde653681616957f96f30b80ab728bcb29ca3831 not found: ID does not exist" Oct 29 22:48:11.215751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.215703 4543 scope.go:122] "RemoveContainer" containerID="6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3" Oct 29 22:48:11.224506 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.224489 4543 scope.go:122] "RemoveContainer" containerID="6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3" Oct 29 22:48:11.224747 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:11.224728 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3\": container with ID starting with 6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3 not found: ID does not exist" containerID="6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3" Oct 29 22:48:11.224811 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.224751 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3"} err="failed to get container status \"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3\": rpc error: code = NotFound desc = could not find container \"6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3\": container with ID starting with 6144f2dfe690a95a66333c4b022774f8c9b8512d24072eb63204cab2b02501b3 not found: ID does not exist" Oct 29 22:48:11.307577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.307530 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpbp9\" (UniqueName: \"kubernetes.io/projected/393e6e93-fcde-4371-adc3-b0a888f3041f-kube-api-access-vpbp9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.307577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.307550 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbv6r\" (UniqueName: \"kubernetes.io/projected/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8-kube-api-access-mbv6r\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.307577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.307565 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393e6e93-fcde-4371-adc3-b0a888f3041f-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.570751 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.570700 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:48:11.583032 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.583009 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-1e74c3-dbd76b8dd-xqkxd"] Oct 29 22:48:11.623923 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.623903 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:48:11.634892 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:11.634872 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-1e74c3-6f47c6c995-758xv"] Oct 29 22:48:12.383917 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:12.383886 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" path="/var/lib/kubelet/pods/393e6e93-fcde-4371-adc3-b0a888f3041f/volumes" Oct 29 22:48:12.384299 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:12.384273 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" path="/var/lib/kubelet/pods/d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8/volumes" Oct 29 22:48:12.709840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:12.709772 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:48:14.596441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:14.596411 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc"] Oct 29 22:48:14.609519 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:14.609495 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8d569e-vz2wc"] Oct 29 22:48:16.384289 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:16.384256 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" path="/var/lib/kubelet/pods/eccf6fc8-ddc5-4fcc-831e-38b6b7aec316/volumes" Oct 29 22:48:17.710045 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:17.709995 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" containerID="cri-o://5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0" gracePeriod=25 Oct 29 22:48:21.292367 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:21.292331 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:22.987705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.987660 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv"] Oct 29 22:48:22.988566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988542 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988568 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988593 4543 cpu_manager.go:414] "containerMap: removing container" podUID="eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" containerName="revoke-keys" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988601 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" containerName="revoke-keys" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988622 4543 cpu_manager.go:414] "containerMap: removing container" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988631 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988654 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" Oct 29 22:48:22.988698 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988661 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988720 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988727 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b613ca-f1c4-4764-b23d-f98afd462fa4" containerName="istio-proxy" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988935 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5097e4-78c1-4d74-9759-d5198b887368" containerName="istio-proxy" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988957 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccf6fc8-ddc5-4fcc-831e-38b6b7aec316" containerName="revoke-keys" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988968 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="393e6e93-fcde-4371-adc3-b0a888f3041f" containerName="payload-pre-processing" Oct 29 22:48:22.989043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.988984 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53b8ee8-d5e3-41d8-a6d1-abb6cc0b69a8" containerName="payload-processing" Oct 29 22:48:22.994974 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:22.994950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.007278 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.007249 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv"] Oct 29 22:48:23.017852 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.017831 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:48:23.017948 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.017832 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:48:23.034441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.034420 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:23.034771 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.034753 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:48:23.110410 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.110385 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.110503 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.110442 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ttp\" (UniqueName: \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.110555 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.110536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.211809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.211774 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.211940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.211861 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ttp\" (UniqueName: \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.211940 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.211906 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.212435 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.212412 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.214390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.214369 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.237662 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.237639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ttp\" (UniqueName: \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\") pod \"maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.310222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.310163 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:23.614578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:23.614553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv"] Oct 29 22:48:23.617525 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:48:23.617500 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd0adb1_d9b2_4ff7_aa2d_b114d11eabb8.slice/crio-f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee WatchSource:0}: Error finding container f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee: Status 404 returned error can't find the container with id f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee Oct 29 22:48:24.293905 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:24.293863 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" event={"ID":"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8","Type":"ContainerStarted","Data":"0ff1456b0fd359e55dfa6c3cbc7b4dd8c29ed2acfc53b0ee126015faf9f8efff"} Oct 29 22:48:24.294360 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:24.293918 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" event={"ID":"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8","Type":"ContainerStarted","Data":"f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee"} Oct 29 22:48:24.340752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:24.340716 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" podStartSLOduration=2.34070469 podStartE2EDuration="2.34070469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:24.340139174 +0000 UTC m=+3966.082069843" watchObservedRunningTime="2026-10-29 22:48:24.34070469 +0000 UTC m=+3966.082635363" Oct 29 22:48:25.301943 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.301905 4543 generic.go:356] "Generic (PLEG): container finished" podID="3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" containerID="0ff1456b0fd359e55dfa6c3cbc7b4dd8c29ed2acfc53b0ee126015faf9f8efff" exitCode=0 Oct 29 22:48:25.302383 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.301996 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" event={"ID":"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8","Type":"ContainerDied","Data":"0ff1456b0fd359e55dfa6c3cbc7b4dd8c29ed2acfc53b0ee126015faf9f8efff"} Oct 29 22:48:25.577586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.577510 4543 scope.go:122] "RemoveContainer" containerID="9736bdde70ddb82a3c6dd43fb34bd3b12648be8bf16dea8a577f0b9b6d679d35" Oct 29 22:48:25.586593 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.586571 4543 scope.go:122] "RemoveContainer" containerID="cdf14737a95d95ea352972210951f95e1342893f3691088aa812c7830d063fd6" Oct 29 22:48:25.598587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.598558 4543 scope.go:122] "RemoveContainer" containerID="9fc17f16b0391c26a1e9aeb0711165a3316df33733eac74b8626d9c6093a32bd" Oct 29 22:48:25.606857 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:25.606832 4543 scope.go:122] "RemoveContainer" containerID="321b4ddf1b296010b93a8652040b103acdca8bcf31a002cd7a37602c2bbf0f84" Oct 29 22:48:26.446511 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.446482 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:26.540628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.540602 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\") pod \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " Oct 29 22:48:26.540789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.540699 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\") pod \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " Oct 29 22:48:26.540789 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.540753 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\" (UniqueName: \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\") pod \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\" (UID: \"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8\") " Oct 29 22:48:26.541008 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.540986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca" pod "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" (UID: "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:26.542856 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.542829 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca" pod "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" (UID: "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:26.542956 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.542894 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp" pod "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" (UID: "3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8"). InnerVolumeSpecName "kube-api-access-l6ttp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:26.642445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.642393 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:26.642445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.642414 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:26.642445 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:26.642423 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6ttp\" (UniqueName: \"kubernetes.io/projected/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8-kube-api-access-l6ttp\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.315710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:27.315675 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" event={"ID":"3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8","Type":"ContainerDied","Data":"f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee"} Oct 29 22:48:27.315998 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:27.315968 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37d6576578db6c57869796957f0211e1d9fc9cf7f6ecffe10dfd6600c54ddee" Oct 29 22:48:27.316235 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:27.316221 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv" Oct 29 22:48:27.743028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:27.742994 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 22:48:27.757209 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:27.757180 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 22:48:27.959955 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:27.959923 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:48:31.291978 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:31.291935 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:33.472462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:33.472430 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 22:48:33.504879 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:33.504845 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 22:48:35.599670 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.599637 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh"] Oct 29 22:48:35.600093 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.600066 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" containerName="revoke-keys" Oct 29 22:48:35.600179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.600094 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" containerName="revoke-keys" Oct 29 22:48:35.600179 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.600178 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" containerName="revoke-keys" Oct 29 22:48:35.603654 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.603631 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.608115 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.608066 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbx6g\" (UniqueName: \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.608263 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.608207 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.608346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.608263 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.625437 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.625411 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh"] Oct 29 22:48:35.708722 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.708694 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbx6g\" (UniqueName: \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.708848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.708755 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.708848 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.708829 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.710002 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.709979 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:35.710199 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.710178 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:48:35.710296 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.710260 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:48:35.710420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.710389 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:48:35.720401 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.720374 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.722336 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.722313 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.738327 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.738264 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbx6g\" (UniqueName: \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\") pod \"maas-api-revoke-sub-d82a6452-9l4dh\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:35.919394 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:35.919339 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:36.221807 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:36.221757 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh"] Oct 29 22:48:36.224945 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:48:36.224915 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27909ffe_216c_4076_9a64_056f6f7002da.slice/crio-e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e WatchSource:0}: Error finding container e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e: Status 404 returned error can't find the container with id e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e Oct 29 22:48:36.372994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:36.372967 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" event={"ID":"27909ffe-216c-4076-9a64-056f6f7002da","Type":"ContainerStarted","Data":"e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e"} Oct 29 22:48:37.382104 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:37.382014 4543 generic.go:356] "Generic (PLEG): container finished" podID="27909ffe-216c-4076-9a64-056f6f7002da" containerID="67efe6a3fc4e42646d6b1ddd9ca8c5819484f50f95bec6bbc56c3f52f31b730c" exitCode=0 Oct 29 22:48:37.382502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:37.382119 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" event={"ID":"27909ffe-216c-4076-9a64-056f6f7002da","Type":"ContainerDied","Data":"67efe6a3fc4e42646d6b1ddd9ca8c5819484f50f95bec6bbc56c3f52f31b730c"} Oct 29 22:48:37.964176 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:37.964148 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:48:38.025858 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.025789 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\" (UniqueName: \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\") pod \"3b2abb55-3000-4980-ad69-1bd1ada616ee\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " Oct 29 22:48:38.026000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.025892 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\") pod \"3b2abb55-3000-4980-ad69-1bd1ada616ee\" (UID: \"3b2abb55-3000-4980-ad69-1bd1ada616ee\") " Oct 29 22:48:38.026264 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.026244 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume" pod "3b2abb55-3000-4980-ad69-1bd1ada616ee" (UID: "3b2abb55-3000-4980-ad69-1bd1ada616ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:38.028043 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.028017 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9" pod "3b2abb55-3000-4980-ad69-1bd1ada616ee" (UID: "3b2abb55-3000-4980-ad69-1bd1ada616ee"). InnerVolumeSpecName "kube-api-access-56gl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:38.127587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.127561 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gl9\" (UniqueName: \"kubernetes.io/projected/3b2abb55-3000-4980-ad69-1bd1ada616ee-kube-api-access-56gl9\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.127587 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.127587 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2abb55-3000-4980-ad69-1bd1ada616ee-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.390404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.390297 4543 generic.go:356] "Generic (PLEG): container finished" podID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerID="5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0" exitCode=0 Oct 29 22:48:38.390404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.390342 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" Oct 29 22:48:38.390404 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.390392 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" event={"ID":"3b2abb55-3000-4980-ad69-1bd1ada616ee","Type":"ContainerDied","Data":"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0"} Oct 29 22:48:38.390979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.390448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz" event={"ID":"3b2abb55-3000-4980-ad69-1bd1ada616ee","Type":"ContainerDied","Data":"ab3984ac4d2a1e107b224582d9dd38cec38ccf884be704f9e7e23e044213e326"} Oct 29 22:48:38.390979 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.390474 4543 scope.go:122] "RemoveContainer" containerID="5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0" Oct 29 22:48:38.403677 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.403660 4543 scope.go:122] "RemoveContainer" containerID="5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0" Oct 29 22:48:38.403988 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:38.403969 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0\": container with ID starting with 5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0 not found: ID does not exist" containerID="5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0" Oct 29 22:48:38.404041 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.403997 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0"} err="failed to get container status \"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0\": rpc error: code = NotFound desc = could not find container \"5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0\": container with ID starting with 5af18607c5327b6de37550bc3fd4fd6f82fe9a2f0ca7a528cf5f32a1984c2bc0 not found: ID does not exist" Oct 29 22:48:38.458222 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.458194 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:48:38.465502 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.465473 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8d569e-55f48d56f9-b84gz"] Oct 29 22:48:38.472692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.472664 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" containerID="cri-o://5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c" gracePeriod=25 Oct 29 22:48:38.505678 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.505647 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" containerID="cri-o://dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a" gracePeriod=25 Oct 29 22:48:38.534834 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.534814 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:38.630652 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.630625 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\") pod \"27909ffe-216c-4076-9a64-056f6f7002da\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " Oct 29 22:48:38.630752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.630692 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\") pod \"27909ffe-216c-4076-9a64-056f6f7002da\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " Oct 29 22:48:38.630800 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.630776 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\" (UniqueName: \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\") pod \"27909ffe-216c-4076-9a64-056f6f7002da\" (UID: \"27909ffe-216c-4076-9a64-056f6f7002da\") " Oct 29 22:48:38.631014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.630989 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca" pod "27909ffe-216c-4076-9a64-056f6f7002da" (UID: "27909ffe-216c-4076-9a64-056f6f7002da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:38.631124 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.631106 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-openshift-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.632826 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.632805 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca" pod "27909ffe-216c-4076-9a64-056f6f7002da" (UID: "27909ffe-216c-4076-9a64-056f6f7002da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:38.632921 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.632851 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g" pod "27909ffe-216c-4076-9a64-056f6f7002da" (UID: "27909ffe-216c-4076-9a64-056f6f7002da"). InnerVolumeSpecName "kube-api-access-lbx6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:38.732013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.731989 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbx6g\" (UniqueName: \"kubernetes.io/projected/27909ffe-216c-4076-9a64-056f6f7002da-kube-api-access-lbx6g\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.732013 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:38.732011 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27909ffe-216c-4076-9a64-056f6f7002da-maas-api-xks-ca\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:39.398588 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.398556 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" Oct 29 22:48:39.399014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.398561 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d82a6452-9l4dh" event={"ID":"27909ffe-216c-4076-9a64-056f6f7002da","Type":"ContainerDied","Data":"e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e"} Oct 29 22:48:39.399014 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.398685 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1533e22970038fab986c27709ea62f0a8010453d32cbdd37b2a92880bb3a04e" Oct 29 22:48:39.471283 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.471255 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 22:48:39.471471 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.471431 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" containerID="cri-o://dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b" gracePeriod=30 Oct 29 22:48:39.908977 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.908946 4543 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:48:39.909180 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:39.909007 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:48:40.387561 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:40.387529 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" path="/var/lib/kubelet/pods/3b2abb55-3000-4980-ad69-1bd1ada616ee/volumes" Oct 29 22:48:42.765798 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:42.765756 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" containerID="cri-o://412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed" gracePeriod=45 Oct 29 22:48:42.776973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:42.776938 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" containerID="cri-o://515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe" gracePeriod=45 Oct 29 22:48:42.932217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:42.932165 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": dial tcp 10.133.0.89:8000: connect: connection refused" Oct 29 22:48:43.022626 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.022570 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:48:43.047509 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.047489 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:48:43.064565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064543 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.064657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064589 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\" (UniqueName: \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.064657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064625 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.064657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064642 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064663 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064693 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064729 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064785 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\" (UniqueName: \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.064809 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064786 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064817 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064910 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\") pod \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\" (UID: \"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7\") " Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064902 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064947 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.064960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065015 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065215 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065152 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065344 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065371 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065394 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065408 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tmp-dir\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065421 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.065779 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.065433 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-model-cache\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.067155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.067112 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "kube-api-access-x5l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:43.067262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.067183 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "kube-api-access-gkkgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:43.067262 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.067213 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.067483 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.067469 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.067532 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.067478 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm" pod "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" (UID: "ebabb8b1-d6fb-436a-975a-1b93ca3c39c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.165797 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.165765 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\") pod \"6d0d9691-09aa-48be-87da-163df31eeef2\" (UID: \"6d0d9691-09aa-48be-87da-163df31eeef2\") " Oct 29 22:48:43.166064 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166046 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5l9q\" (UniqueName: \"kubernetes.io/projected/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-kube-api-access-x5l9q\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.166198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166069 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0d9691-09aa-48be-87da-163df31eeef2-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.166198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166109 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-home\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.166198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166123 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkgh\" (UniqueName: \"kubernetes.io/projected/6d0d9691-09aa-48be-87da-163df31eeef2-kube-api-access-gkkgh\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.166198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166136 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-tls-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.166198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.166147 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.167864 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.167840 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm" pod "6d0d9691-09aa-48be-87da-163df31eeef2" (UID: "6d0d9691-09aa-48be-87da-163df31eeef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.267333 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.267299 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0d9691-09aa-48be-87da-163df31eeef2-dshm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.424390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.424315 4543 generic.go:356] "Generic (PLEG): container finished" podID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerID="412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed" exitCode=0 Oct 29 22:48:43.424390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.424356 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" Oct 29 22:48:43.424609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.424399 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" event={"ID":"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7","Type":"ContainerDied","Data":"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed"} Oct 29 22:48:43.424609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.424437 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69" event={"ID":"ebabb8b1-d6fb-436a-975a-1b93ca3c39c7","Type":"ContainerDied","Data":"72919435179f6d08431bee30f28b9adc146bd451a976dee07d237da12bcf3522"} Oct 29 22:48:43.424609 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.424458 4543 scope.go:122] "RemoveContainer" containerID="412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed" Oct 29 22:48:43.425885 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.425854 4543 generic.go:356] "Generic (PLEG): container finished" podID="6d0d9691-09aa-48be-87da-163df31eeef2" containerID="515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe" exitCode=0 Oct 29 22:48:43.426024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.425884 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" event={"ID":"6d0d9691-09aa-48be-87da-163df31eeef2","Type":"ContainerDied","Data":"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe"} Oct 29 22:48:43.426024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.425905 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" Oct 29 22:48:43.426024 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.425922 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x" event={"ID":"6d0d9691-09aa-48be-87da-163df31eeef2","Type":"ContainerDied","Data":"28814f005afa6ea2d2df9b03a91770178d6609edbb23be7867e8afd0be8dac96"} Oct 29 22:48:43.435244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.435228 4543 scope.go:122] "RemoveContainer" containerID="412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed" Oct 29 22:48:43.435525 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:43.435502 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed\": container with ID starting with 412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed not found: ID does not exist" containerID="412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed" Oct 29 22:48:43.435623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.435535 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed"} err="failed to get container status \"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed\": rpc error: code = NotFound desc = could not find container \"412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed\": container with ID starting with 412f61874220ef79d4db8ee1499d1b17b4383067b15da76b8e1657bee7fcb2ed not found: ID does not exist" Oct 29 22:48:43.435623 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.435552 4543 scope.go:122] "RemoveContainer" containerID="515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe" Oct 29 22:48:43.443829 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.443810 4543 scope.go:122] "RemoveContainer" containerID="515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe" Oct 29 22:48:43.444136 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:43.444054 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe\": container with ID starting with 515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe not found: ID does not exist" containerID="515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe" Oct 29 22:48:43.444136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.444096 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe"} err="failed to get container status \"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe\": rpc error: code = NotFound desc = could not find container \"515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe\": container with ID starting with 515a137e8fa64432d0d7beaeb047e58eb37496058c616f007ca8cb3afd568dfe not found: ID does not exist" Oct 29 22:48:43.482308 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.482288 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 22:48:43.487496 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.487478 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-2-w5-91520c-kserve-5766fc8cf-5bh69"] Oct 29 22:48:43.525420 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.525399 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 22:48:43.531147 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.531127 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/distinct-w5-91520c-kserve-d77754bcb-6nw2x"] Oct 29 22:48:43.900000 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:43.899976 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:44.391006 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.390971 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" path="/var/lib/kubelet/pods/6d0d9691-09aa-48be-87da-163df31eeef2/volumes" Oct 29 22:48:44.391434 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.391417 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" path="/var/lib/kubelet/pods/ebabb8b1-d6fb-436a-975a-1b93ca3c39c7/volumes" Oct 29 22:48:44.718365 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.718341 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 22:48:44.778661 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778635 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.778787 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778686 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.778846 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778805 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.778887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778859 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.778939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778887 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.778939 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778913 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.779066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778965 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:44.779066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.778980 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.779066 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779038 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.779258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779090 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\") pod \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\" (UID: \"91175bfe-bfc9-4318-9e64-25bdfec8ab33\") " Oct 29 22:48:44.779381 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779352 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:44.779468 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779269 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:44.779536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779422 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.779536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779424 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:44.779536 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.779527 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:44.782190 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.782143 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:44.782485 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.782448 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:44.782801 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.782777 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:44.783368 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.783344 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw" pod "91175bfe-bfc9-4318-9e64-25bdfec8ab33" (UID: "91175bfe-bfc9-4318-9e64-25bdfec8ab33"). InnerVolumeSpecName "kube-api-access-spggw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:44.880578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880557 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880578 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880577 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880586 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880595 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880603 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880612 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880620 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spggw\" (UniqueName: \"kubernetes.io/projected/91175bfe-bfc9-4318-9e64-25bdfec8ab33-kube-api-access-spggw\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.880710 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.880629 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91175bfe-bfc9-4318-9e64-25bdfec8ab33-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:44.904462 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:44.904438 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:45.441351 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.441317 4543 generic.go:356] "Generic (PLEG): container finished" podID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerID="dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b" exitCode=0 Oct 29 22:48:45.441501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.441372 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" Oct 29 22:48:45.441501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.441401 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" event={"ID":"91175bfe-bfc9-4318-9e64-25bdfec8ab33","Type":"ContainerDied","Data":"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b"} Oct 29 22:48:45.441501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.441456 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw" event={"ID":"91175bfe-bfc9-4318-9e64-25bdfec8ab33","Type":"ContainerDied","Data":"2bd5c2aae535edfc9c08ebe27a57cf33dbb1196c7782a2638bdc7062e10a0d9f"} Oct 29 22:48:45.441501 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.441479 4543 scope.go:122] "RemoveContainer" containerID="dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b" Oct 29 22:48:45.453136 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.453120 4543 scope.go:122] "RemoveContainer" containerID="dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b" Oct 29 22:48:45.453403 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:45.453383 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b\": container with ID starting with dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b not found: ID does not exist" containerID="dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b" Oct 29 22:48:45.453455 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.453411 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b"} err="failed to get container status \"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b\": rpc error: code = NotFound desc = could not find container \"dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b\": container with ID starting with dd9cd0ff4bdde00b2cf3e6d3bc9c59f6623492d338a659e179fb903a088b601b not found: ID does not exist" Oct 29 22:48:45.514696 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.514670 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 22:48:45.527037 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:45.527012 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-whqtw"] Oct 29 22:48:46.383887 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:46.383839 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" path="/var/lib/kubelet/pods/91175bfe-bfc9-4318-9e64-25bdfec8ab33/volumes" Oct 29 22:48:47.181752 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:47.181713 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 22:48:47.182015 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:47.181949 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" containerID="cri-o://ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3" gracePeriod=30 Oct 29 22:48:48.192705 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:48.192672 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fmz5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:48:48.193142 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:48.192723 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:48:51.051107 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.051049 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm"] Oct 29 22:48:51.072859 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.072821 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddad8b8e-bsgsm"] Oct 29 22:48:51.073021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.072881 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh"] Oct 29 22:48:51.073021 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.072920 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926c3146-wz4zh"] Oct 29 22:48:51.319791 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.319711 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv"] Oct 29 22:48:51.324584 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:51.324557 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-91520c-8tlkv"] Oct 29 22:48:52.385716 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.385680 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c000f3-3fe1-4b68-8c17-c7d566919a78" path="/var/lib/kubelet/pods/11c000f3-3fe1-4b68-8c17-c7d566919a78/volumes" Oct 29 22:48:52.386174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.386104 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8" path="/var/lib/kubelet/pods/3cd0adb1-d9b2-4ff7-aa2d-b114d11eabb8/volumes" Oct 29 22:48:52.386493 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.386477 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24c5eab-d7a5-442e-9938-aa32c80c48b3" path="/var/lib/kubelet/pods/f24c5eab-d7a5-442e-9938-aa32c80c48b3/volumes" Oct 29 22:48:52.436840 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.436820 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 22:48:52.484334 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.484309 4543 generic.go:356] "Generic (PLEG): container finished" podID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerID="ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3" exitCode=0 Oct 29 22:48:52.484441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.484345 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" Oct 29 22:48:52.484441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.484389 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" event={"ID":"e3c78f57-8e7a-438e-997a-a3df95e5a7ff","Type":"ContainerDied","Data":"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3"} Oct 29 22:48:52.484441 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.484439 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k" event={"ID":"e3c78f57-8e7a-438e-997a-a3df95e5a7ff","Type":"ContainerDied","Data":"9bd8361f929f99696a9605506dbb60782f65dba2884357db377d37fbd0d6d6cc"} Oct 29 22:48:52.484577 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.484461 4543 scope.go:122] "RemoveContainer" containerID="ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3" Oct 29 22:48:52.493498 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.493480 4543 scope.go:122] "RemoveContainer" containerID="ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3" Oct 29 22:48:52.493770 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:52.493750 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3\": container with ID starting with ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3 not found: ID does not exist" containerID="ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3" Oct 29 22:48:52.493819 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.493777 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3"} err="failed to get container status \"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3\": rpc error: code = NotFound desc = could not find container \"ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3\": container with ID starting with ed5f5b096fe74e12de0ee5d662119a275995b4d615875cfd1594eb87148eecc3 not found: ID does not exist" Oct 29 22:48:52.548582 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548561 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548599 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548692 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548641 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548774 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548737 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548825 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548783 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548862 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548833 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548867 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548914 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.548981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548940 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:52.548981 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.548946 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\") pod \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\" (UID: \"e3c78f57-8e7a-438e-997a-a3df95e5a7ff\") " Oct 29 22:48:52.549258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549008 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:52.549258 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549141 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:52.549384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549261 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:52.549384 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549307 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:52.549490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549385 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istiod-ca-cert\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.549490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549405 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.549490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549421 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-data\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.549490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549434 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-workload-certs\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.549490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.549447 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-credential-socket\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.551288 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.551267 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:52.551377 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.551307 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:52.551514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.551494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:52.551514 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.551501 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm" pod "e3c78f57-8e7a-438e-997a-a3df95e5a7ff" (UID: "e3c78f57-8e7a-438e-997a-a3df95e5a7ff"). InnerVolumeSpecName "kube-api-access-fl7xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:52.650824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.650799 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-podinfo\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.650824 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.650821 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-token\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.650953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.650831 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7xm\" (UniqueName: \"kubernetes.io/projected/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-kube-api-access-fl7xm\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.650953 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.650840 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c78f57-8e7a-438e-997a-a3df95e5a7ff-istio-envoy\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:52.835217 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.835186 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 22:48:52.842198 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:52.842178 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fmz5k"] Oct 29 22:48:53.899759 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:53.899722 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:54.384007 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:54.383974 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" path="/var/lib/kubelet/pods/e3c78f57-8e7a-438e-997a-a3df95e5a7ff/volumes" Oct 29 22:48:54.904549 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:54.904509 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:57.959502 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:57.959469 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:48:58.745429 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.745378 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 22:48:58.770913 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.770892 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 22:48:58.795969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.795934 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\") pod \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " Oct 29 22:48:58.796109 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.795999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\" (UniqueName: \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\") pod \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\" (UID: \"0433cc1f-5fe7-4f71-9726-2c36f02abcc0\") " Oct 29 22:48:58.796342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.796320 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume" pod "0433cc1f-5fe7-4f71-9726-2c36f02abcc0" (UID: "0433cc1f-5fe7-4f71-9726-2c36f02abcc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:58.798329 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.798295 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77" pod "0433cc1f-5fe7-4f71-9726-2c36f02abcc0" (UID: "0433cc1f-5fe7-4f71-9726-2c36f02abcc0"). InnerVolumeSpecName "kube-api-access-r6t77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:58.897174 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.897153 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\" (UniqueName: \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\") pod \"f04497cd-0a50-440b-a735-3d63585b76f5\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " Oct 29 22:48:58.897290 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.897196 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\") pod \"f04497cd-0a50-440b-a735-3d63585b76f5\" (UID: \"f04497cd-0a50-440b-a735-3d63585b76f5\") " Oct 29 22:48:58.897427 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.897408 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:58.897470 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.897434 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6t77\" (UniqueName: \"kubernetes.io/projected/0433cc1f-5fe7-4f71-9726-2c36f02abcc0-kube-api-access-r6t77\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:58.897603 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.897588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume" pod "f04497cd-0a50-440b-a735-3d63585b76f5" (UID: "f04497cd-0a50-440b-a735-3d63585b76f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:58.899286 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.899244 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k" pod "f04497cd-0a50-440b-a735-3d63585b76f5" (UID: "f04497cd-0a50-440b-a735-3d63585b76f5"). InnerVolumeSpecName "kube-api-access-ftd2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:58.998741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.998688 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftd2k\" (UniqueName: \"kubernetes.io/projected/f04497cd-0a50-440b-a735-3d63585b76f5-kube-api-access-ftd2k\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:58.998741 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:58.998712 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04497cd-0a50-440b-a735-3d63585b76f5-plugins-config-volume\") on node \"ip-10-0-128-100.ec2.internal\" DevicePath \"\"" Oct 29 22:48:59.529608 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.529574 4543 generic.go:356] "Generic (PLEG): container finished" podID="f04497cd-0a50-440b-a735-3d63585b76f5" containerID="dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a" exitCode=0 Oct 29 22:48:59.529736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.529626 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" Oct 29 22:48:59.529736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.529670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" event={"ID":"f04497cd-0a50-440b-a735-3d63585b76f5","Type":"ContainerDied","Data":"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a"} Oct 29 22:48:59.529736 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.529720 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf" event={"ID":"f04497cd-0a50-440b-a735-3d63585b76f5","Type":"ContainerDied","Data":"2a6af3470058eadf003a719eb591c6089c0e08286d57c47f34f3a1d11d803e6a"} Oct 29 22:48:59.529916 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.529750 4543 scope.go:122] "RemoveContainer" containerID="dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a" Oct 29 22:48:59.531200 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.531174 4543 generic.go:356] "Generic (PLEG): container finished" podID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerID="5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c" exitCode=0 Oct 29 22:48:59.531332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.531224 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" Oct 29 22:48:59.531332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.531262 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" event={"ID":"0433cc1f-5fe7-4f71-9726-2c36f02abcc0","Type":"ContainerDied","Data":"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c"} Oct 29 22:48:59.531332 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.531318 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4" event={"ID":"0433cc1f-5fe7-4f71-9726-2c36f02abcc0","Type":"ContainerDied","Data":"cd508aa1dc2a228088a27bdce8c03d1ca9f45dbe8b5773409fd186206d4b1262"} Oct 29 22:48:59.540473 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.540455 4543 scope.go:122] "RemoveContainer" containerID="dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a" Oct 29 22:48:59.540811 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:59.540776 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a\": container with ID starting with dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a not found: ID does not exist" containerID="dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a" Oct 29 22:48:59.540937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.540822 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a"} err="failed to get container status \"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a\": rpc error: code = NotFound desc = could not find container \"dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a\": container with ID starting with dfb0ee4c3ea6f49cd562cd4b69c52ba84a8deee91f4c2dd6bbc368913804c64a not found: ID does not exist" Oct 29 22:48:59.540937 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.540841 4543 scope.go:122] "RemoveContainer" containerID="5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c" Oct 29 22:48:59.550482 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.550464 4543 scope.go:122] "RemoveContainer" containerID="5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c" Oct 29 22:48:59.550744 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:48:59.550729 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c\": container with ID starting with 5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c not found: ID does not exist" containerID="5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c" Oct 29 22:48:59.550803 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.550754 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c"} err="failed to get container status \"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c\": rpc error: code = NotFound desc = could not find container \"5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c\": container with ID starting with 5bd614ea85792eb9840c65a111bd54ae6cfd612f276a9abfac8890a9410e189c not found: ID does not exist" Oct 29 22:48:59.590512 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.590481 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 22:48:59.599314 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.599263 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-91520c-5876f9b455-bldtf"] Oct 29 22:48:59.638782 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.638756 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 22:48:59.644098 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:48:59.644061 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-91520c-5bbdf686b4-h2jw4"] Oct 29 22:49:00.384028 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:00.383992 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" path="/var/lib/kubelet/pods/0433cc1f-5fe7-4f71-9726-2c36f02abcc0/volumes" Oct 29 22:49:00.384465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:00.384400 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" path="/var/lib/kubelet/pods/f04497cd-0a50-440b-a735-3d63585b76f5/volumes" Oct 29 22:49:01.534566 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:01.534533 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s2sn9_016e6143-39a6-45b3-ac25-cce513d56269/kuadrant-console-plugin/0.log" Oct 29 22:49:20.985220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:20.985181 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0adbf973-nkpzt_19e5d7dc-f015-427f-9472-247ac056297b/revoke-keys/0.log" Oct 29 22:49:21.176214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:21.176183 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3522aa13-c4x5s_d8630ee4-7d00-4f36-95a4-634e3127c263/revoke-keys/0.log" Oct 29 22:49:21.369216 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:21.369148 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-69764cd9-8knrp_2c8f8750-de5c-4795-9c7e-97ebda53b478/revoke-keys/0.log" Oct 29 22:49:21.559090 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:21.559045 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7cab32cf-28pgm_3dbb372f-c0cf-4df3-b895-a121136542f2/revoke-keys/0.log" Oct 29 22:49:21.737241 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:21.737212 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-853c13b1-kmgx4_a8a71bc1-6db0-46d2-bf70-82e33ba3cead/revoke-keys/0.log" Oct 29 22:49:21.920312 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:21.920287 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b0fdac1e-p94bf_a20b8b47-7514-4bf4-84e2-deb5a2ddcc5e/revoke-keys/0.log" Oct 29 22:49:22.116631 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:22.116572 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b713f27b-ptp8f_149ec293-c313-4360-b728-250da7da7388/revoke-keys/0.log" Oct 29 22:49:22.483421 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:22.483391 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d82a6452-9l4dh_27909ffe-216c-4076-9a64-056f6f7002da/revoke-keys/0.log" Oct 29 22:49:22.655823 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:22.655802 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-effe9389-4mjc4_db29b465-7dcc-424a-a70a-a684b260b775/revoke-keys/0.log" Oct 29 22:49:24.992969 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:24.992939 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s2sn9_016e6143-39a6-45b3-ac25-cce513d56269/kuadrant-console-plugin/0.log" Oct 29 22:49:25.680051 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.680022 4543 scope.go:122] "RemoveContainer" containerID="fba26e27297cd81a98d2ab87d6daad78e0caf69c3f89f800ea724b190d110f7e" Oct 29 22:49:25.689211 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.689191 4543 scope.go:122] "RemoveContainer" containerID="5058159716c7b1167636fa1129d25c67e37211afb70f351ee838859feacd57c9" Oct 29 22:49:25.697949 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.697931 4543 scope.go:122] "RemoveContainer" containerID="5e6a0e708b956a9f658a1c6e8eb2940b189e4f3635f8f2bd4ca844d7a02224e8" Oct 29 22:49:25.709950 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.709931 4543 scope.go:122] "RemoveContainer" containerID="beb39cb7e0059427d884b129e928bcafb80995a64c8014441917769ecb67b6fd" Oct 29 22:49:25.718641 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.718618 4543 scope.go:122] "RemoveContainer" containerID="f55f28b06250856faec7f1892785d43d493756ac4377614e27876a0f8a2ab47b" Oct 29 22:49:25.726769 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.726751 4543 scope.go:122] "RemoveContainer" containerID="022413909f3e80108241640df4fbb427e886cbf40da3a8e2e294c2e43c2877c8" Oct 29 22:49:25.734726 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.734697 4543 scope.go:122] "RemoveContainer" containerID="cfaad37594b564b17869a27ef19894fbd1eff72656b48bc555ccce06e72e09f5" Oct 29 22:49:25.742419 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:25.742401 4543 scope.go:122] "RemoveContainer" containerID="0e88eb314942f506d4bd614ed6295a92284d402e5568d0c75e7387fc5b986eae" Oct 29 22:49:27.032450 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:27.032420 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v8ds5_c271c3c4-d946-4bce-afe7-1df9c7364966/discovery/0.log" Oct 29 22:49:27.418712 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:27.418626 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-vq2rr_a17112fb-d40c-4e98-903f-7100f859628b/kube-auth-proxy/0.log" Oct 29 22:49:27.960772 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:49:27.960740 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:49:29.792287 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:29.792251 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-xbmp9_83e19b48-3028-4018-88ec-ce30df35e7bd/main/0.log" Oct 29 22:49:29.983192 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:29.983160 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-f4xp2_40f8925d-8433-499f-9523-93d536c176f2/main/0.log" Oct 29 22:49:30.788591 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:30.788557 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-n6x4r_e92f01a9-e0b3-4c27-9ed2-63e20b27df7a/main/0.log" Oct 29 22:49:30.968941 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:30.968902 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6vv7l_e386a9e9-223d-4c2d-96c2-148d940e4044/main/0.log" Oct 29 22:49:47.576244 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:47.576215 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2tdpb_29210f25-02b2-48e3-9d5b-24fbf755276a/global-pull-secret-syncer/0.log" Oct 29 22:49:48.702942 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:48.702905 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-znvhf_a4f0a27b-37b7-4dc0-932d-913d95a9d0e4/konnectivity-agent/0.log" Oct 29 22:49:48.802086 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:48.802046 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-100.ec2.internal_2787fae5ca7f0b39f010401de9cba34b/haproxy/0.log" Oct 29 22:49:56.034636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:56.034599 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4"] Oct 29 22:49:56.047855 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:56.047830 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe9389-4mjc4"] Oct 29 22:49:56.383504 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:56.383427 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db29b465-7dcc-424a-a70a-a684b260b775" path="/var/lib/kubelet/pods/db29b465-7dcc-424a-a70a-a684b260b775/volumes" Oct 29 22:49:57.628971 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:57.628936 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s2sn9_016e6143-39a6-45b3-ac25-cce513d56269/kuadrant-console-plugin/0.log" Oct 29 22:49:57.961836 ip-10-0-128-100 kubenswrapper[4543]: E1029 22:49:57.961799 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jpxgr" containerName="extractor" Oct 29 22:49:58.988065 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988033 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv"] Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988404 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988418 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988432 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988438 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988445 4543 cpu_manager.go:414] "containerMap: removing container" podUID="27909ffe-216c-4076-9a64-056f6f7002da" containerName="revoke-keys" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988450 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27909ffe-216c-4076-9a64-056f6f7002da" containerName="revoke-keys" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988459 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988466 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988483 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988487 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988499 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988503 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988517 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988521 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988528 4543 cpu_manager.go:414] "containerMap: removing container" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" Oct 29 22:49:58.988499 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988533 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988591 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c78f57-8e7a-438e-997a-a3df95e5a7ff" containerName="istio-proxy" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988601 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="91175bfe-bfc9-4318-9e64-25bdfec8ab33" containerName="istio-proxy" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988608 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0d9691-09aa-48be-87da-163df31eeef2" containerName="main" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988615 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04497cd-0a50-440b-a735-3d63585b76f5" containerName="payload-pre-processing" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988621 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="27909ffe-216c-4076-9a64-056f6f7002da" containerName="revoke-keys" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988626 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0433cc1f-5fe7-4f71-9726-2c36f02abcc0" containerName="payload-processing" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988633 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2abb55-3000-4980-ad69-1bd1ada616ee" containerName="payload-pre-processing" Oct 29 22:49:58.989461 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.988640 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebabb8b1-d6fb-436a-975a-1b93ca3c39c7" containerName="main" Oct 29 22:49:58.991706 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:58.991685 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.012491 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.012464 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv"] Oct 29 22:49:59.023657 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.023636 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"kube-root-ca.crt\"" Oct 29 22:49:59.023770 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.023718 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"openshift-service-ca.crt\"" Oct 29 22:49:59.059984 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.059964 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gbwp5\"/\"default-dockercfg-ldfvl\"" Oct 29 22:49:59.087342 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.087315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwp8\" (UniqueName: \"kubernetes.io/projected/f1c1525d-ac0f-499b-a902-11c70ef11a15-kube-api-access-hrwp8\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.087451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.087351 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-lib-modules\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.087451 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.087376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-proc\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.087565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.087454 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-podres\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.087565 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.087526 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-sys\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188390 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188356 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwp8\" (UniqueName: \"kubernetes.io/projected/f1c1525d-ac0f-499b-a902-11c70ef11a15-kube-api-access-hrwp8\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-lib-modules\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188449 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-proc\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188523 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188475 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-podres\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188539 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-sys\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188572 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-lib-modules\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-proc\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188686 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188678 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-sys\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.188837 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.188680 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1c1525d-ac0f-499b-a902-11c70ef11a15-podres\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.214738 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.214710 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwp8\" (UniqueName: \"kubernetes.io/projected/f1c1525d-ac0f-499b-a902-11c70ef11a15-kube-api-access-hrwp8\") pod \"perf-node-gather-daemonset-czbkv\" (UID: \"f1c1525d-ac0f-499b-a902-11c70ef11a15\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.305071 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.305010 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.613781 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.613755 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv"] Oct 29 22:49:59.616292 ip-10-0-128-100 kubenswrapper[4543]: W1029 22:49:59.616260 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1c1525d_ac0f_499b_a902_11c70ef11a15.slice/crio-22df9e90b559188b98058115658fd39127baf6756de80148ec8098dc488a2d0f WatchSource:0}: Error finding container 22df9e90b559188b98058115658fd39127baf6756de80148ec8098dc488a2d0f: Status 404 returned error can't find the container with id 22df9e90b559188b98058115658fd39127baf6756de80148ec8098dc488a2d0f Oct 29 22:49:59.880973 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.880913 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" event={"ID":"f1c1525d-ac0f-499b-a902-11c70ef11a15","Type":"ContainerStarted","Data":"1728367eb559c210e610ae02adfda67819e1de03435c542f4986f74784c23a8f"} Oct 29 22:49:59.881114 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.881095 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:49:59.881167 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.881138 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" event={"ID":"f1c1525d-ac0f-499b-a902-11c70ef11a15","Type":"ContainerStarted","Data":"22df9e90b559188b98058115658fd39127baf6756de80148ec8098dc488a2d0f"} Oct 29 22:49:59.929056 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:49:59.929004 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" podStartSLOduration=1.9289887700000001 podStartE2EDuration="1.92898877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:59.92771231 +0000 UTC m=+4061.669642982" watchObservedRunningTime="2026-10-29 22:49:59.92898877 +0000 UTC m=+4061.670919444" Oct 29 22:50:02.484305 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:02.484276 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wj5m_41636e36-7a36-4e98-920e-8a1126f6ccea/node-exporter/0.log" Oct 29 22:50:02.601195 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:02.601165 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wj5m_41636e36-7a36-4e98-920e-8a1126f6ccea/kube-rbac-proxy/0.log" Oct 29 22:50:02.718220 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:02.718191 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wj5m_41636e36-7a36-4e98-920e-8a1126f6ccea/init-textfile/0.log" Oct 29 22:50:03.966636 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:03.966605 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/prometheus/0.log" Oct 29 22:50:04.077155 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.077111 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/config-reloader/0.log" Oct 29 22:50:04.185397 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.185359 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/thanos-sidecar/0.log" Oct 29 22:50:04.294994 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.294941 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/kube-rbac-proxy-web/0.log" Oct 29 22:50:04.416590 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.416570 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/kube-rbac-proxy/0.log" Oct 29 22:50:04.553755 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.553705 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/kube-rbac-proxy-thanos/0.log" Oct 29 22:50:04.680628 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:04.680605 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0297c7a4-dfb0-4b05-a7cd-2d77f9a6c3ba/init-config-reloader/0.log" Oct 29 22:50:05.551326 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:05.551294 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/thanos-query/0.log" Oct 29 22:50:05.667650 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:05.667618 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/kube-rbac-proxy-web/0.log" Oct 29 22:50:05.795012 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:05.794988 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/kube-rbac-proxy/0.log" Oct 29 22:50:05.896214 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:05.896160 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-czbkv" Oct 29 22:50:05.909120 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:05.909068 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/prom-label-proxy/0.log" Oct 29 22:50:06.033718 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:06.033697 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/kube-rbac-proxy-rules/0.log" Oct 29 22:50:06.145121 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:06.145063 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f85cdbb4f-87l69_68b24797-fe50-4957-a874-de6bf3feb83a/kube-rbac-proxy-metrics/0.log" Oct 29 22:50:08.037595 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:08.037561 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s"] Oct 29 22:50:08.050242 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:08.050214 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3522aa13-c4x5s"] Oct 29 22:50:08.383101 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:08.383020 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8630ee4-7d00-4f36-95a4-634e3127c263" path="/var/lib/kubelet/pods/d8630ee4-7d00-4f36-95a4-634e3127c263/volumes" Oct 29 22:50:15.392586 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:15.392548 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p75bp_332737e0-8635-454a-942a-2ba3ca23e966/dns/0.log" Oct 29 22:50:15.517400 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:15.517376 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p75bp_332737e0-8635-454a-942a-2ba3ca23e966/kube-rbac-proxy/0.log" Oct 29 22:50:16.016465 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:16.016437 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2k6br_7672c73e-69be-484f-84ae-12c26a104e59/dns-node-resolver/0.log" Oct 29 22:50:16.091651 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:16.091628 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v8ds5_c271c3c4-d946-4bce-afe7-1df9c7364966/discovery/0.log" Oct 29 22:50:16.307490 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:16.307423 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-vq2rr_a17112fb-d40c-4e98-903f-7100f859628b/kube-auth-proxy/0.log" Oct 29 22:50:17.468346 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:17.468314 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56c44567bf-hdhbj_dd24f7c5-67b5-49fe-8691-06b90349e21d/registry/0.log" Oct 29 22:50:17.846219 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:17.846138 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkxjk_fad6f194-bef0-48b9-bf0a-725194f1703a/node-ca/0.log" Oct 29 22:50:18.640822 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:18.640791 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v8ds5_c271c3c4-d946-4bce-afe7-1df9c7364966/discovery/0.log" Oct 29 22:50:18.665185 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:18.665161 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v8ds5_c271c3c4-d946-4bce-afe7-1df9c7364966/discovery/0.log" Oct 29 22:50:18.751533 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:18.751510 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-vq2rr_a17112fb-d40c-4e98-903f-7100f859628b/kube-auth-proxy/0.log" Oct 29 22:50:18.787986 ip-10-0-128-100 kubenswrapper[4543]: I1029 22:50:18.787959 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698579f7b7-vq2rr_a17112fb-d40c-4e98-903f-7100f859628b/kube-auth-proxy/0.log"